First of all, we need to add our first Fragment at the beginning, we should do it in the onCreate() method of our Activity:

if (null == savedInstanceState) {
    getSupportFragmentManager().beginTransaction()
      .addToBackStack("fragmentA")
      .replace(R.id.container, FragmentA.newInstance(), "fragmentA")
      .commit();
}

Next, we need to manage our backstack. The easiest way is using a function added in our activity that is used for all FragmentTransactions.

public void replaceFragment(Fragment fragment, String tag) {
    //Get current fragment placed in container
    Fragment currentFragment = getSupportFragmentManager().findFragmentById(R.id.container);

    //Prevent adding same fragment on top
    if (currentFragment.getClass() == fragment.getClass()) {
        return;
    }

    //If fragment is already on stack, we can pop back stack to prevent stack infinite growth
    if (getSupportFragmentManager().findFragmentByTag(tag) != null) {
        getSupportFragmentManager().popBackStack(tag, FragmentManager.POP_BACK_STACK_INCLUSIVE);
    }

    //Otherwise, just replace fragment
    getSupportFragmentManager()
            .beginTransaction()
            .addToBackStack(tag)
            .replace(R.id.container, fragment, tag)
            .commit();
}

Finally, we should override onBackPressed() to exit the application when going back from the last Fragment available in the backstack.

@Override
public void onBackPressed() {
    int fragmentsInStack = getSupportFragmentManager().getBackStackEntryCount();
    if (fragmentsInStack > 1) { // If we have more than one fragment, pop back stack
        getSupportFragmentManager().popBackStack();
    } else if (fragmentsInStack == 1) { // Finish activity, if only one fragment left, to prevent leaving empty screen
        finish();
    } else {
        super.onBackPressed();
    }
}

Execution in activity:

replaceFragment(FragmentB.newInstance(), "fragmentB");

Execution outside activity (assuming MainActivity is our activity):

((MainActivity) getActivity()).replaceFragment(FragmentB.newInstance(), "fragmentB");