So I've figured more details about this. Beside doing what I described able, there's also issues with ModPagespeed and the themes. ModPagespeed cause a lot of issues with plugins and apps.
So I disabled it and it works.
ModPagespeedDisallow "*domain.com/directory/*"
My question is what if you have more than one directories that you want to this disable? Do you just add another line like this?
ModPagespeedDisallow "*domain.com/directory/*"
ModPagespeedDisallow "*domain.com/directory2/*"
ModPagespeedDisallow "*domain.com/directory3/dir1/3*"
Or is there a better way of doing this that cleaner?