Using Brown v. Board of Education is bad legal precedent.
Extending racial segregation rulings to the issue of same-sex marriage would result in other Constitutional infringements.
The result of Brown v. Board was that schools had to racially integrate their facilities in order to provide education. While there are other legal arguments that may support legal same-sex marriages, this one in particular is build upon the use of facilities in providing services. The logical extention of using Brown v. Board is that churches must integrate same-sex couples into their marriage services. This violates the free exercise clause of the first amendment. If you want to use legal precedent to argue in favor of same-sex marriage, use another ruling.