We install the AU and VST plugins in the user’s Library folder NOT the system Library folder.
The AU plugin is at location:
/Library/Audio/Plug-Ins/Components
And the VSTs are at locations:
/Library/Audio/Plug-Ins/VST
and
/Library/Audio/Plug-Ins/VST3
The AAX plugin is at:
/Library/Application Support/Avid/Audio/Plug-Ins
If you need to open these folders use Finder's "Go" menu and choose "Go to folder…" then type in the paths as written above.