To display checkbox options in vertical direction, layout property is used.
Example:
<apex:selectRadio value=“{!searchCategory}”
layout=“pageDirection” >
layout=“pageDirection” >
<apex:selectOption itemValue=“Member__c”
itemlabel=“Member”/>
itemlabel=“Member”/>
<apex:selectOption itemValue=“Blog__c”
itemlabel=“Blog”/>
itemlabel=“Blog”/>
<apex:selectOption itemValue=“Photo__c”
itemlabel=“Photo”/>
itemlabel=“Photo”/>
</apex:selectRadio>
Cheers!!!