Prerequisites

• You need OpenMeetings 1.9.x or later for this, OpenMeetings 1.8.x does not have those options.
• Install OpenMeetings according to the install instructions and check that it runs without problems
• Rename the existing keystore file red5/conf/keystore.jmx to red5/conf/keystore.bak
• Rename the existing truststore file red5/conf/truststore.jmx to red5/conf/truststore.bak

Create Keystore from the scratch

1. Create a new keystore and key, use the same password for both:
   
   

   keytool -keysize 2048 -genkey -alias red5 -keyalg RSA -keystore red5/conf/keystore.jks
   Enter keystore password:
   Re-enter new password:
   What is your first and last name?
   [Unknown]:  <your hostname, e.g demo.openmeetings.de>
   What is the name of your organizational unit?
   [Unknown]:  Dev
   What is the name of your organization?
   [Unknown]:  OpenMeetings
   What is the name of your City or Locality?
   [Unknown]:  Henderson
   What is the name of your State or Province?
   [Unknown]:  Nevada
   What is the two-letter country code for this unit?
   [Unknown]:  US
   Is CN=demo.openmeetings.de, OU=Dev, O=OpenMeetings, L=Henderson, ST=Nevada, C=US correct?
   [no]:  yes
   Enter key password for <red5>
   

2. Generate a CSR:

   keytool -certreq -keyalg RSA -alias red5 -file red5.csr -keystore red5/conf/keystore.jks

3. Submit CSR to your CA of choice and receive a signed certificate
4. Import your chosen CA's root certificate into the keystore (may need to download it from their site - make sure to get the root CA and not the intermediate one):

   keytool -import -alias root -keystore red5/conf/keystore.jks -trustcacerts -file root.crt

   (note: you may receive a warning that the certificate already exists in the system wide keystore - import anyway)
5. Import the intermediate certificate(s) you normally receive with the certificate:

   keytool -import -alias intermed -keystore red5/conf/keystore.jks -trustcacerts -file intermediate.crt

6. Import the certificate you received:

   keytool -import -alias red5 -keystore red5/conf/keystore.jks -trustcacerts -file demo.openmeetings.de.crt

7. Please NOTE according to this http://javarevisited.blogspot.com/2012/09/difference-between-truststore-vs-keyStore-Java-SSL.html guide you can split keystore and truststore (OPTIONAL you might just copy keystore to truststore)
8. Create additional certificate as described above. Add this certificate to the following keystores: red5/conf/keystore.screen and red5/conf/keystore.jks. (This step is required to be able to use screen-sharing web application, you can copy "main" keystore while testing)

Create Keystore using existing key-pair

Prerequisites

• Create CA's root certificate: ca.crt
• Create self-signed server certificate: red5.crt Common Name (CN) while creating certificate should be assign to FQDN of your site, for example - vkc.company.com
• Rename the existing keystore file red5/conf/keystore.jmx to red5/conf/keystore.bak
• Rename the existing truststore file red5/conf/truststore.jmx to red5/conf/truststore.bak

Steps for OM server

1. Export existing keys into PKCS12 format:

   cd _folder_with_certificates_
   openssl pkcs12 -export -in red5.crt -inkey red5.key -out red5.p12 -name red5 -certfile ca.crt
   

2. Import resulting red5.p12 into keystore:

   keytool -importkeystore -srcstorepass changeit -srckeystore red5.p12 -srcstoretype PKCS12 -deststorepass changeit -destkeystore /opt/red5/conf/keystore.jks -alias red5
   

   This command creates the keystore.jks with password changeit (may be any password)
3. Import your CA's root certificate into the keystore:

   keytool -import -alias root -keystore /opt/red5/conf/keystore.jks -keystorepass changeit -trustcacerts -file ca.crt
   

4. copy keystore to truststore

   cp /opt/red5/conf/keystore.jks /opt/red5/conf/truststore.jks
   

5. Change passwords in /opt/red5/conf/red5.properties

   rtmps.keystorepass=changeit
   rtmps.truststorepass=changeit
   jmx.keystorepass=changeit
   

6. Set up SSL according SSL for the web interface section.
7. Set up RTMPS according Tunneling RTMPS or Native RTMPS sections.
8. Restart OM service or whole OM server. Now OM server is ready to accept SSL-connections.

Steps for client machines

Windows specific steps are marked with (**)

1. (**)Add the ca.crt certificate as trusted into the CA's list on all windows PC at the system level (certmgr.msc applet)
2. Add the ca.crt certificate as trusted into Chrome/Firefox if it used
3. Add the ca.crt certificate into ${JAVA_HOME}/lib/security/cacerts with keytool utility on a PC that organizes screensharing

   (**)
   For example, Java Version 8 Update 144 has been installed on Windows machine by default location path.
   Copy ca.crt to a folder : c:\ca.crt
   Start command line interface as administrator:
   runas /user:AdminAccount cmd.exe
   Enter admin password
   Enter to bin directory of java:
   cd "c:\Program Files\Java\jre1.8.0_144\bin"
   Import ca.crt into keystore:
   keytool -import -alias root -keystore ..\lib\security\cacerts -file c:\ca.crt
   Enter the password to keystore: changeit
   

Tunneling RTMPS

1. HTTPS need to be enabled otherwise tunneling will not work (it can be set up using frontend nginx/apache as well)
2. In Administration->Configuration set
   
   

   					flash.secure = true
   					flash.secure.proxy = none
   

3. Restart red5 and try to connect - your connection should now be made via RTMPS (close port 1935 to be sure)

Native RTMPS

1. Default RTMPS port is 8443, you can change it by editing red5/conf/red5.properties and change the port here: rtmps.port=8443
   Please set rtmps.keystorepass=password and rtmps.truststorepass=password (password = password you set on your new keystore(s))
   Additionally you need to set rtmps.screen.keystorepass=screenpassword (screenpassword = password you set on your keystore for screen-sharing application)
2. Edit red5/conf/red5-core.xml file:
   UNComment RTMPS section
3. In Administration->Configuration set
   
   

   					flash.secure = true
   					flash.secure.proxy = best
   

4. Restart red5 and try to connect - your connection should now be made via RTMPS (close port 1935 to be sure)