Back when I was doing Salesforce consulting, the first thing I would check in a new client's org is how they are handling test fixtures.
Usually I would find one of two things:

• SObjects would be created inside the tests themselves:
  

    static testMethod void myUnitTest(){
        Account testAccount = new Account();
        testAccount.Name = 'Test Account';
        testAccount.Website = 'Website';
        insert testAccount
        //run tests
    }
  

• Some kind of TestingUtils.cls with EVERYTHING related to creating test fixtures
  

    @isTest
    public class TestingUtils{
        public static List<Account> buildAccounts( Integer numOfAccounts ){
            List<Account> testAccounts = new List<Account>();
            for( Integer i = 0; i < numOfAccounts; i++ ){
                testAccounts.add( new Account() );
            }
            return testAccounts;
        }
  
        //700 lines later...
  
        public static List<Task> buildTasks( Integer numOfTasks ){
            List<Task> testTasks = new List<Task>();
            for( Integer i = 0; i < numOfTasks; i++ ){
                testTasks.add( new Task() );
            }
            return testTasks;
        }
  

These are both bad ideas because they create an unmaintainable mess (which I learned first hand by doing both).
Test fixtures coded in the test will break the second you make a new required field, which will require you to find and fix every fixture.
The TestingUtils class at least keeps everything in one place, but it is rife with it's own issues:

• It will continue to grow into a massive god class as you add more fixtures and all of the weird edge cases( e.g. buildAccountWithContacts, buildAccountWithParentAccount etc.)
• Not DRY at all: the same basic pattern is used in each of the methods: build n objects and return the list of objects.
• Inevitable merge conflicts this causes when several developers all need to modify this file in each of their feature branches

I didn't really have a solution nor the opportunity to present one. I was hesitant to try establishing a new standard in an org that wasn't really mine and wante to respect the house rules. In orgs that didn't have any test fixture standards, I would try to break up the factory methods into different classes, but a TestingUtils.cls would inevitably appear and I would give in, especially since I didn't solve the DRY problem and the last thing we needed was two standards to further confuse things.
When I became the Lead Developer at Apto, I quickly fell into old habits. While working with another developer a few months later, I lamented to him about my increasingly growing TestSObjectFactory.cls and how I wish I could make it more modular. With a confused look, he asked me why I don't just do it. It then dawned on me that this was our house and we could make the rules as we saw fit (If you couldn't tell, I love xckd).
So we implemented a SObjectFactory abstract class that other factories could extend from. The common logic was handled by the abstract class and each extended class can handle all of the unique logic and hold any required fields.
For Example:

    @isTest
    public class AccountFactory extends SObjectFactory{
        public SObject buildRecordWithData(){
            return new Account( Name = 'Test Account ' + counter );
        }
    }

Then you can call all of the abstracted methods:

    AccountFactory factory = new AccountFactory();
    List<Account> accountInstances = factory.buildRecords(10); 
    List<Account> insertedAccounts = factory.buildAndInsertRecords(10);
    Account accountInstance = (Account)factory.buildRecord();
    Account insertedAccount = (Account)factory.buildAndInsertRecord();

You can also pass in a map of fields to values:

    final String EXPECTED_FIELD_VALUE = 'www.example.com';
    Map<Schema.SObjectField, Object> fieldtoValue
        = new Map<Schema.SObjectField, Object>{ Account.Website =>
                                                    EXPECTED_FIELD_VALUE }; 
    AccountFactory factory = new AccountFactory();
    factory.setFieldToValueMapping( fieldtoValue );
    Account accountInstance = factory.buildRecord();
    System.assertEquals( EXPECTED_FIELD_VALUE, accountInstance.Website );

The code is much DRYer, no more repeated logic, and less chance for merge conflicts.
But aren't you worried about class explosion?
Class explosion is much better than dealing with a god class. Each SObject is very different and thus should be treated as so. The toilet factory doesn't belong in the same place as the candy factory, that would just be weird and confusing.
Feel free to check out the code and use it and your org, it's using the MIT License. But the real moral of the story is, reflect on what you are coding. If you foresee the mess you are creating, then you should probably stop and think and it over.