From Red5Tutorials

Create Server Keystore:

keytool -genkey -alias red5server -keyalg RSA -validity 36500 \ 
-keystore conf/jmx.keystore -storepass password -keypass password \ 
-dname "CN=Dan Rossi, OU=IT, O=Red5, L=Sydney,S=NSW, C=AU"

Verify Keystore:

keytool -list -v -keystore conf/jmx.keystore -storepass password

Export Self Signed Cert:

keytool -export -alias red5server -keystore conf/jmx.keystore \ 
-file conf/red5server.cer -storepass password

Create Client Truststore:

keytool -genkey -alias red5client -keyalg RSA -validity 36500 -keystore conf/jmx.truststore \
-storepass trustword -keypass trustword -dname "CN=Dan Rossi, OU=IT, O=Red5, L=Sydney,S=NSW, C=AU"

Verify Client Truststore:

keytool -list -v -keystore conf/jmx.truststore -storepass trustword

Import Server Cert into TrustStore:

keytool -import -file conf/red5server.cer -keystore conf/jmx.truststore -storepass trustword -noprompt

Edit conf/red5-common.xml, put true next to enableSsl, and set the rmi port next to rmiAdapterPort.

  <bean id="jmxAgent" class="org.red5.server.jmx.JMXAgent" init-method="init">
              <property name="enableRmiAdapter" value="true" />
              <property name="rmiAdapterPort" value="9999"/>
               <property name="enableSsl" value="true"/>
               <property name="remoteAccessProperties" value="conf/access.properties"/>
               <property name="remotePasswordProperties" value="conf/password.properties"/>
               <property name="enableHtmlAdapter" value="false" />
               <property name="htmlAdapterPort" value="8082"/>
  </bean>

Edit conf/password.properties and put in a clear text password for authentication

red5user changeme

Edit conf/access.properties and put in the readwrite rights for the user

red5user readwrite

Start RMI Registry (unless it is going to started programatically)

rmiregistry -J-Djava.security.manager -J-Djava.security.policy=conf/red5.policy \
-J-Djavax.net.ssl.trustStore=conf/jmx.truststore -J-Djavax.net.ssl.trustStorePassword=trustword 9999 &

Start Red5:

With these args

-Djava.security.manager 
-Djava.security.policy=conf/red5.policy 
-Dcom.sun.management.jmxremote 
-Djavax.net.ssl.keyStore=conf/jmx.keystore
-Djavax.net.ssl.keyStorePassword=password

Example:

java -Djava.security.manager -Djava.security.policy=conf/red5.policy -Dcom.sun.management.jmxremote \ 
-Djavax.net.ssl.keyStore=conf/jmx.keystore -Djavax.net.ssl.keyStorePassword=password -cp red5.jar:conf:$CLASSPATH org.red5.server.Standalone

Start Jconsole:

jconsole -J-Djava.security.manager -J-Djava.security.policy=conf/red5.policy \
-J-Djavax.net.ssl.trustStore=conf/jmx.truststore -J-Djavax.net.ssl.trustStorePassword=trustword \
-J-Djava.security.debug=ssl service:jmx:rmi:///jndi/rmi://host:9999/red5