To determine the normal form in DBMS, analyze the functional dependencies within the table and progressively decompose it into smaller, more normalized relations while avoiding data anomalies and redundancy.