Maltfield Log/2018 Q1

From Open Source Ecology
Jump to navigation Jump to search

My work log from the year 2018 Quarter 1. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.

See Also

  1. Maltfield_Log
  2. User:Maltfield
  3. Special:Contributions/Maltfield

Sun May 20, 2018

  1. I arrived to FeF! I finally met Marcin & Catarina in-person and got a tour of the workshop, seedhomes, etc.
  2. My purpose for this visit is primarily driven by the need to ensure that Marcin has access to our live keepass (which necessitates having ssh access to our server) and as well as a local copy of his ssh key, personal keepass, and our shared ose keepass. We may also migrate the wiki tomorrow, depending on the status of its validation

Fri May 18, 2018

  1. I consolidated all our modsecurity whitelists from all our wordpress sites into one long, numerically sorted list, then I added this to all our wordpress sites' vhost configs. This will prevent false-positive 403 issues that have been fixed on one wordpress site from cropping up on another. It's not ideal, but it's a pragmatic compromise.
[root@hetzner2 conf.d]# date
Fri May 18 14:29:10 UTC 2018
[root@hetzner2 conf.d]# pwd
/etc/httpd/conf.d
[root@hetzner2 conf.d]# grep -A 3 '<LocationMatch "/(wp-admin|ose-hidden-login)/">' * | grep 'SecRuleRemoveById' | tr " " "\n" | sort -un | grep -vi SecRuleRemoveById | tr "\n" " "
200003 200004 950001 950109 950120 950901 958008 958030 958051 958056 958057 959070 959072 959073 960015 960017 960020 960024 960335 960904 960915 970901 973300 973301 973304 973306 973316 973327 973329 973330 973331 973332 973333 973334 973335 973336 973337 973338 973344 973347 981172 981173 981231 981240 981242 981243 981244 981245 981246 981248 981253 981257 981317 981318 981319 [root@hetzner2 conf.d]# 
[root@hetzner2 conf.d]# 
    1. this was applied to the following vhost files
[root@hetzner2 conf.d]# grep -irl '<LocationMatch "/(wp-admin|ose-hidden-login)/">' *
000-www.opensourceecology.org.conf
00-fef.opensourceecology.org.conf
00-oswh.opensourceecology.org.conf
00-seedhome.openbuildinginstitute.org.conf
00-www.openbuildinginstitute.org.conf
staging.opensourceecology.org.conf
[root@hetzner2 conf.d]# 
    1. for records, here is what the files had before the change
[root@hetzner2 conf.d]# grep -irA 3 '<LocationMatch "/(wp-admin|ose-hidden-login)/">' * | grep 'SecRuleRemoveById'
000-www.opensourceecology.org.conf-                     SecRuleRemoveById 960015 960024 960904 960015 960017 970901 950109 981172 981231 981245 973338 973306 950901 981317 959072 981257 981243 958030 973300 973304 973335 973333 973316 200004 973347 981319 981240 973301 973344 960335 960020 950120 959073 981244 981248 981253 973334 973332 981242 981246 958057 958056 973327 973337 950001 973336 958051 973331 973330 959070 958008 973329 960024
00-fef.opensourceecology.org.conf-                      SecRuleRemoveById 960015 960024 960904 960015 960017 970901 950109 981172 981231 981245 973338 973306 950901 981317 959072 981257 981243 958030 973300 973304 973335 973333 973316 200004 973347 981319 981240 973301 973344 960335 960020 950120 959073 981244 981248 981253 973334 973332 981242 981246 960915 200003
00-oswh.opensourceecology.org.conf-                     SecRuleRemoveById 960015 960024 960904 960015 960017 970901 950109 981172 981231 981245 973338 973306 950901 981317 959072 981257 981243 958030 973300 973304 973335 973333 973316 200004 973347 981319 981240 973301 973344 960335 960020 950120 959073 981244 981248 981253 973334 973332 981242 981246
00-seedhome.openbuildinginstitute.org.conf-                        SecRuleRemoveById 960015 981173 960024 960904 960015 960017 970901 950109 981172 981231 981245 973338 973306 950901 981317 959072 981257 981243 958030 973300 973304 973335 973333 973316 200004 973347 981319 981240 973301 973344 960335 960020 950120
00-www.openbuildinginstitute.org.conf-                        SecRuleRemoveById 960015 960024 960904 960015 960017 970901 950109 981172 981231 981245 973338 973306 950901 981317 959072 981257 981243 958030 973300 973304 973335 973333 973316 200004 973347 981319 981240 973301 973344 960335 960020 950120 959073 981244 981248 981253 973334 973332 981242 981246 981318
staging.opensourceecology.org.conf-                     SecRuleRemoveById 960015 960024 960904 960015 960017 970901 950109 981172 981231 981245 973338 973306 950901 981317 959072 981257 981243 958030 973300 973304 973335 973333 973316 200004 973347 981319 981240 973301 973344 960335 960020 950120 959073 981244 981248 981253 973334 973332 981242 981246 958057 958056 973327 973337 950001 973336 958051 973331 973330 959070
[root@hetzner2 conf.d]# 
    1. I actually just wrapped up this new mod_security whitelist rules into a new file at /etc/httpd/conf.d/mod_security.wordpress.include . This way, when we add one, we add it to all sites.
      1. I intentionally did not do this with the other common wordpress blocks, such as blocking of '.git' dirs, blocking 'wp-login.php', etc as I don't want someone to comment-out the include in attempt to debug a mod_security issue, and suddenly disable these other critical security blocks which never false-positive like mod_security. Also, this mod_security stuff actually needs to be updated so the include file helps. The other stuff is essentially static.
  1. I also added a block that prevents files from being executed by php that have been placed into the uploads dir
   # don't execute any php files inside the uploads directory                                                                                                      
   <LocationMatch "/wp-content/uploads/">                                                                                                                          
	  php_flag engine off                                                                                                                                          
   </LocationMatch>                                                                                                                                                
   <LocationMatch "/wp-content/uploads/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">                                                                                     
	  Order Deny,Allow                                                                                                                                             
	  Deny from All                                                                                                                                                
   </LocationMatch>   
  1. Marcin said we should migrate the wiki Tuesday pending validation.
    1. I spent some time formally documenting all the wiki changes here http://opensourceecology.org/wiki/CHG-2018-05-22
    2. I added a banner notice message to the prod wiki site with "$wgSiteNotice" to inform our users of the upcoming maintenance window, and I linked to the CHG above
  2. Marcin sent me another 403 forbidden false-positive. I whitelisted 950907 = "generic" / "system command injection" attack and asked him to try again
  3. attempted to update the "3d printer workshop" page and I immediately got some modsecurity false-positives, which I whitelisted
    1. 981256 sqli
    2. 981249 sqli
  4. Marcin sent me another string that was triggering modsec false-positives. The fix was to whitliest these rules:
    1. 958413 xss

Thr May 17, 2018

  1. Marcin forwarded me a security alert from Dreamhost that our server had been sending spam from the 'ose_marcin' account. Note that recently we got an alert from them about the 'ose_community' account on that server, which had been running drupal. I changed that user's password & shutdown the vhost already. Now this is a distinct account! But trying to investigate this damn incident on a shared server without root is like trying to weld with a soldering iron. I sent them an email asking many questions & for more information about what happened.

Wed May 16, 2018

  1. My request to join the meetecho-janus google group yesterday was approved
  2. I posted a thread to the meetecho-janus google group asking for janus security best-practices https://groups.google.com/forum/#!topic/meetecho-janus/0Vx_Vl0hmwU
  3. I updated my git issue. Lorenzo updated their site, and I was able to confirm that the issue occurs there too. https://github.com/meetecho/janus-gateway/issues/1233
  4. I tried to research ICE hardening, but again the searches for security around webrtc lead to marketing guides talking about how secure it is for the client
  5. while I wait for a response from the janus community on my hardening thread, I began to research how we can administer the videoroom. Specifically, we need to be able to select which participants can become a publisher instead of just a subscriber. This is something OpenTok does well for clients like the MLB, but that shit ain't open.
  6. there are some janus configuration options regarding authentication with the api, but that appears to be all-or-nothing auth. There doesn't appear to be anything that would specifically allow a subscriber to escalate themselves to becoming a producer. https://janus.conf.meetecho.com/docs/auth.html
  7. found this which says "you can control who can join, but you can not control his activities after join." https://groups.google.com/forum/#!searchin/meetecho-janus/videoroom$20publish$20authentication%7Csort:date/meetecho-janus/TJivBoiOXA0/KaqrfKx0AwAJ
  8. so we may have to write a modified version of the videoroom.
  9. or we can just password protect the whole videoroom, and then just capture the videoroom somehow and rebroadcast it through another subscribe-only channel similar to how youtube live works.
    1. I posted this question in all of its ignorance here; we'll see what happens.. https://janus.conf.meetecho.com/docs/auth.html

Tue May 15, 2018

  1. the main dev behind Janus is Lorenzo Miniero, and he's the one that responded to my git issue in <5 minutes. I found an interview with him about the Open Source Janus Gateway here after reviewing his LInkedIn https://www.linkedin.com/pulse/meet-meetecho-janus-gateway-fabian-bernhard
  2. he also came from The University of Naples Federico II in Naples, Italy--which is where a lot of these WebRTC experts appear to have originated..
  3. I applied to write messages on the 'meetecho-janus' mailing list. after I'm approved, I'll ask the community if there's any guides on how to harden Janus' configuration or security best-practices. For example, file permissions, hardened configuration options for the each of the config files (main, transport, plugins, etc)

Mon May 14, 2018

  1. continuing to debug why jangouts' text chat didn't work. The text room demo in janus also failed, and it said that it sent the data with "data-channels"
  2. I could not find out what the initial configure options were when I compiled janus (`janus --version` doesn't list it)
  3. I tried to reconfigure janus, this time explicitly setting '--enable-data-channels'. It failed with an error from libusrsctp.
[root@ip-172-31-28-115 janus-gateway]# ./configure --enable-data-channels
...
checking for srtp_crypto_policy_set_aes_gcm_256_16_auth in -lsrtp2... yes
checking for usrsctp_finish in -lusrsctp... no
configure: error: libusrsctp not found. See README.md for installation instructions or use --disable-data-channels
[root@ip-172-31-28-115 janus-gateway]# 
  1. the main janus gateway git README explicitly lists usrsctp as a dependency, stating "(only needed if you are interested in Data Channels)". It links to the usrsctp github here https://github.com/sctplab/usrsctp
  2. the usrsctp github doesn't provide instructions for centos7. It states that it's tested for FreeBSD, Ubuntu, Windows, & Mac.
  3. I attempted to compile it manually, as internet searches suggested that it's not in any yum repo.
pushd /root/sandbox
git clone https://github.com/sctplab/usrsctp
pushd usrsctp
./bootstrap
./configure && make && sudo make install
popd
popd
  1. trying to reconfigure lists "DataChannels support: yes", which I confirmed was previously "no"
config.status: executing libtool commands

libsrtp version:           2.x
SSL/crypto library:        OpenSSL
DTLS set-timeout:          not available
DataChannels support:      yes
Recordings post-processor: no
TURN REST API client:      yes
Doxygen documentation:     no
Transports:
	REST (HTTP/HTTPS):     yes
	WebSockets:            no
	RabbitMQ:              no
	MQTT:                  no
	Unix Sockets:          yes
Plugins:
	Echo Test:             yes
	Streaming:             yes
	Video Call:            yes
	SIP Gateway (Sofia):   no
	SIP Gateway (libre):   no
	NoSIP (RTP Bridge):    yes
	Audio Bridge:          no
	Video Room:            yes
	Voice Mail:            no
	Record&Play:           yes
	Text Room:             yes
	Lua Interpreter:       yes
Event handlers:
	Sample event handler:  yes
	RabbitMQ event handler:no
JavaScript modules:        no

If this configuration is ok for you, do a 'make' to start building Janus. A 'make install' will install Janus and its plugins to the specified prefix. Finally, a 'make configs' will install some sample configuration files too (something you'll only want to do the first time, though).

[root@ip-172-31-28-115 janus-gateway]# 
  1. I recompiled & deployed with `make && make install`, then restarted janus. Unfortunately, I have the same issue
Session: 8126727588102204
Handle: 3978092954853617
Processing POST data (application/json) (310 bytes)...
[transports/janus_http.c:janus_http_handler:1248]   -- Data we have now (310 bytes)
[transports/janus_http.c:janus_http_handler:1170] Processing HTTP POST request on /janus/8126727588102204/3978092954853617...
[transports/janus_http.c:janus_http_handler:1223]  ... parsing request...
Session: 8126727588102204
Handle: 3978092954853617
Processing POST data (application/json) (0 bytes)...
[transports/janus_http.c:janus_http_handler:1253] Done getting payload, we can answer
{"janus":"message","body":{"request":"ack"},"transaction":"RvFOm1M7roLf","jsep":{"type":"answer","sdp":"v=0\r\no=- 6893769308065182494 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=msid-semantic: WMS\r\nm=application 0 DTLS/SCTP 5000\r\nc=IN IP4 0.0.0.0\r\na=mid:data\r\na=sctpmap:5000 webrtc-datachannel 1024\r\n"}}
Forwarding request to the core (0x7f00ec000ca0)
Got a Janus API request from janus.transport.http (0x7f00ec000ca0)
Transport task pool, serving request
[3978092954853617] There's a message for JANUS TextRoom plugin
[3978092954853617] Remote SDP:
v=0
o=- 6893769308065182494 2 IN IP4 127.0.0.1
s=-
t=0 0
a=msid-semantic: WMS
m=application 0 DTLS/SCTP 5000
c=IN IP4 0.0.0.0
a=mid:data
a=sctpmap:5000 webrtc-datachannel 1024
[3978092954853617] Audio has NOT been negotiated, Video has NOT been negotiated, SCTP/DataChannels have NOT been negotiated
[WARN] [3978092954853617] Skipping disabled/unsupported media line...
[ERR] [janus.c:janus_process_incoming_request:1193] Error processing SDP
[RvFOm1M7roLf] Returning Janus API error 465 (Error processing SDP)
  1. I stumbled on yet another open source webrtc SFU based on node = Mediasoup https://mediasoup.org/about/
  2. I also found a formal description of SFUs in RFC7667 https://tools.ietf.org/html/rfc7667#section-3.7
  3. doh! It looks like my "./configure" today didn't have the "--prefix /opt/janus" as I used per the README in their github, so my test above was using the old version https://github.com/meetecho/janus-gateway
[root@ip-172-31-28-115 janus-gateway]# LD_LIBRARY_PATH=/usr/lib && /opt/janus/bin/janus --version
Janus commit: d8da250294cbdc193252ce059ef281ba0e2ff5bd
Compiled on:  Fri May  4 00:11:11 UTC 2018

janus 0.4.0
[root@ip-172-31-28-115 janus-gateway]# LD_LIBRARY_PATH=/usr/local/lib && janus --version
Janus commit: d8da250294cbdc193252ce059ef281ba0e2ff5bd
Compiled on:  Mon May 14 14:25:03 UTC 2018

janus 0.4.0
[root@ip-172-31-28-115 janus-gateway]# which janus
/usr/local/bin/janus
[root@ip-172-31-28-115 janus-gateway]# 
  1. I did the compile again, and here's the result
[root@ip-172-31-28-115 janus-gateway]# LD_LIBRARY_PATH=/usr/lib && /opt/janus/bin/janus --version
Janus commit: d8da250294cbdc193252ce059ef281ba0e2ff5bd
Compiled on:  Mon May 14 15:45:32 UTC 2018

janus 0.4.0
[root@ip-172-31-28-115 janus-gateway]# 
  1. I had issues starting janus, which were resolved by adding '/usr/local/lib' to '/etc/ld.so.conf.d/janus.conf' and running `ldconfig`
  2. unfortunately, I have the same issue. note that the sdp message is distinct in chrome & firefox
    1. here's the sdp message in chromium (per the janus server logs on highest verbosity)
Session: 6375072996036015
Handle: 1600250370708259
Processing POST data (application/json) (0 bytes)...
[transports/janus_http.c:janus_http_handler:1253] Done getting payload, we can answer
{"janus":"message","body":{"request":"ack"},"transaction":"LMu5bjOxNA1q","jsep":{"type":"answer","sdp":"v=0\r\no=- 8310479853867794458 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=msid-semantic: WMS\r\nm=application 0 DTLS/SCTP 5000\r\nc=IN IP4 0.0.0.0\r\na=mid:data\r\na=sctpmap:5000 webrtc-datachannel 1024\r\n"}}
Forwarding request to the core (0x7f1538000f80)
Got a Janus API request from janus.transport.http (0x7f1538000f80)
Transport task pool, serving request
[1600250370708259] There's a message for JANUS TextRoom plugin
[1600250370708259] Remote SDP:
v=0
o=- 8310479853867794458 2 IN IP4 127.0.0.1
s=-
t=0 0
a=msid-semantic: WMS
m=application 0 DTLS/SCTP 5000
c=IN IP4 0.0.0.0
a=mid:data
a=sctpmap:5000 webrtc-datachannel 1024
[1600250370708259] Audio has NOT been negotiated, Video has NOT been negotiated, SCTP/DataChannels have NOT been negotiated
[WARN] [1600250370708259] Skipping disabled/unsupported media line...
[ERR] [janus.c:janus_process_incoming_request:1193] Error processing SDP
[LMu5bjOxNA1q] Returning Janus API error 465 (Error processing SDP)
    1. and here's the same thing when the client running the textroomtest demo is firefox instead
Session: 654029176767371
Handle: 6994444633419195
Processing POST data (application/json) (0 bytes)...
[transports/janus_http.c:janus_http_handler:1253] Done getting payload, we can answer
{"janus":"message","body":{"request":"ack"},"transaction":"HtiY4UW9UZDF","jsep":{"type":"answer","sdp":"v=0\r\no=mozilla...THIS_IS_SDPARTA-50.1.0 4746781219317630708 0 IN IP4 0.0.0.0\r\ns=-\r\nt=0 0\r\na=fingerprint:sha-256 CC:E1:78:4E:53:A6:A7:9F:DB:06:B4:4C:68:E8:FB:8B:B3:C7:56:C8:8D:B8:F0:A8:B4:5F:E4:45:FF:1B:39:7B\r\na=group:BUNDLE\r\na=ice-options:trickle\r\na=msid-semantic:WMS *\r\nm=application 0 DTLS/SCTP 5000\r\nc=IN IP4 0.0.0.0\r\na=inactive\r\na=sctpmap:5000 rejected 0\r\n"}}
Forwarding request to the core (0x7f155001b920)
Got a Janus API request from janus.transport.http (0x7f155001b920)
Transport task pool, serving request
[6994444633419195] There's a message for JANUS TextRoom plugin
[6994444633419195] Remote SDP:
v=0
o=mozilla...THIS_IS_SDPARTA-50.1.0 4746781219317630708 0 IN IP4 0.0.0.0
s=-
t=0 0
a=fingerprint:sha-256 CC:E1:78:4E:53:A6:A7:9F:DB:06:B4:4C:68:E8:FB:8B:B3:C7:56:C8:8D:B8:F0:A8:B4:5F:E4:45:FF:1B:39:7B
a=group:BUNDLE
a=ice-options:trickle
a=msid-semantic:WMS *
m=application 0 DTLS/SCTP 5000
c=IN IP4 0.0.0.0
a=inactive
a=sctpmap:5000 rejected 0
[6994444633419195] Audio has NOT been negotiated, Video has NOT been negotiated, SCTP/DataChannels have NOT been negotiated
[6994444633419195] Fingerprint (global) : sha-256 CC:E1:78:4E:53:A6:A7:9F:DB:06:B4:4C:68:E8:FB:8B:B3:C7:56:C8:8D:B8:F0:A8:B4:5F:E4:45:FF:1B:39:7B
[WARN] [6994444633419195] Skipping disabled/unsupported media line...
[ERR] [janus.c:janus_process_incoming_request:1193] Error processing SDP
[HtiY4UW9UZDF] Returning Janus API error 465 (Error processing SDP)
  1. well, when starting janus, there is a warning stating that Data Channels support is *not* compiled
[root@ip-172-31-28-115 janus-gateway]# /opt/janus/bin/janus
...
[WARN] The libsrtp installation does not support AES-GCM profiles
Fingerprint of our certificate: D2:B9:31:8F:DF:24:D8:0E:ED:D2:EF:25:9E:AF:6F:B8:34:AE:53:9C:E6:F3:8F:F2:64:15:FA:E8:7F:53:2D:38
[WARN] Data Channels support not compiled
[WARN] Event handlers support disabled
Plugins folder: /opt/janus/lib/janus/plugins
Loading plugin 'libjanus_recordplay.so'...
  1. ugh, I forgot `make clean` before the `make && make install`. adding that step got me much further! When I loaded the text room, it prompted me for my username (before it just hung indefinitely). Unfortunately, after this popped-up, I got a notification in the browser that we lost connection to the janus gateway. Hopping back to the server, I saw a Segmentation Fault :(
Session: 4989268396723854
Handle: 45723605327998
Processing POST data (application/json) (0 bytes)...
[transports/janus_http.c:janus_http_handler:1253] Done getting payload, we can answer
{"janus":"message","body":{"request":"ack"},"transaction":"9tdbOIEVuv9q","jsep":{"type":"answer","sdp":"v=0\r\no=- 8019385961591100028 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE data\r\na=msid-semantic: WMS\r\nm=application 9 DTLS/SCTP 5000\r\nc=IN IP4 0.0.0.0\r\nb=AS:30\r\na=ice-ufrag:MNDb\r\na=ice-pwd:8F39sum8obXhdVgCLhNhUVLo\r\na=fingerprint:sha-256 D5:D6:25:60:4D:24:9A:37:79:55:4C:B2:F4:99:B0:69:DE:A5:F4:F0:4C:72:CD:67:5C:0F:A9:17:BB:E1:FC:00\r\na=setup:active\r\na=mid:data\r\na=sctpmap:5000 webrtc-datachannel 1024\r\n"}}
Forwarding request to the core (0x7fa31c004620)
Got a Janus API request from janus.transport.http (0x7fa31c004620)
Transport task pool, serving request
[45723605327998] There's a message for JANUS TextRoom plugin
[45723605327998] Remote SDP:
v=0
o=- 8019385961591100028 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE data
a=msid-semantic: WMS
m=application 9 DTLS/SCTP 5000
c=IN IP4 0.0.0.0
b=AS:30
a=ice-ufrag:MNDb
a=ice-pwd:8F39sum8obXhdVgCLhNhUVLo
a=fingerprint:sha-256 D5:D6:25:60:4D:24:9A:37:79:55:4C:B2:F4:99:B0:69:DE:A5:F4:F0:4C:72:CD:67:5C:0F:A9:17:BB:E1:FC:00
a=setup:active
a=mid:data
a=sctpmap:5000 webrtc-datachannel 1024 
[45723605327998] Audio has NOT been negotiated, Video has NOT been negotiated, SCTP/DataChannels have been negotiated
[45723605327998] Parsing SCTP candidates (stream=1)...
[45723605327998] ICE ufrag (local):   MNDb
[45723605327998] ICE pwd (local):     8F39sum8obXhdVgCLhNhUVLo
[45723605327998] Fingerprint (local) : sha-256 D5:D6:25:60:4D:24:9A:37:79:55:4C:B2:F4:99:B0:69:DE:A5:F4:F0:4C:72:CD:67:5C:0F:A9:17:BB:E1:FC:00
[45723605327998] DTLS setup (local):  active
[45723605327998] Setting accept state (DTLS server)
[45723605327998] Data Channel mid: data
Got a sctpmap attribute: 5000 webrtc-datachannel 1024
[45723605327998]   -- ICE Trickling is supported by the browser, waiting for remote candidates...
 -------------------------------------------
  >> Anonymized
 -------------------------------------------
Creating plugin result...
Sending Janus API response to janus.transport.http (0x7fa31c004620)
Got a Janus API response to send (0x7fa31c004620)
Destroying plugin result...
[45723605327998] Sending event to transport...
Sending event to janus.transport.http (0x7fa31c003ce0)
Got a Janus API event to send (0x7fa31c003ce0)
  >> Pushing event: 0 (Success)
[transports/janus_http.c:janus_http_handler:1137] Got a HTTP POST request on /janus/4989268396723854/45723605327998...
[transports/janus_http.c:janus_http_handler:1138]  ... Just parsing headers for now...
[transports/janus_http.c:janus_http_headers:1690] Host: jangouts.opensourceecology.org:8089
[transports/janus_http.c:janus_http_headers:1690] Connection: keep-alive
[transports/janus_http.c:janus_http_headers:1690] Content-Length: 227
[transports/janus_http.c:janus_http_headers:1690] accept: application/json, text/plain, */*
[transports/janus_http.c:janus_http_headers:1690] Origin: https://jangouts.opensourceecology.org
[transports/janus_http.c:janus_http_headers:1690] User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36
[transports/janus_http.c:janus_http_headers:1690] content-type: application/json
[transports/janus_http.c:janus_http_headers:1690] Referer: https://jangouts.opensourceecology.org/textroomtest.html
[transports/janus_http.c:janus_http_headers:1690] Accept-Encoding: gzip, deflate, br
[transports/janus_http.c:janus_http_headers:1690] Accept-Language: en-US,en;q=0.8
[transports/janus_http.c:janus_http_handler:1170] Processing HTTP POST request on /janus/4989268396723854/45723605327998...
[transports/janus_http.c:janus_http_handler:1223]  ... parsing request...
Session: 4989268396723854
Handle: 45723605327998
Processing POST data (application/json) (227 bytes)...
[transports/janus_http.c:janus_http_handler:1248]   -- Data we have now (227 bytes)
[transports/janus_http.c:janus_http_handler:1170] Processing HTTP POST request on /janus/4989268396723854/45723605327998...
[transports/janus_http.c:janus_http_handler:1223]  ... parsing request...
Session: 4989268396723854
Handle: 45723605327998
Processing POST data (application/json) (0 bytes)...
[transports/janus_http.c:janus_http_handler:1253] Done getting payload, we can answer
{"janus":"trickle","candidate":{"candidate":"candidate:201398067 1 udp 2122260223 10.137.2.17 46853 typ host generation 0 ufrag MNDb network-id 1 network-cost 50","sdpMid":"data","sdpMLineIndex":0},"transaction":"JpDJKwdL9Rj4"}
Forwarding request to the core (0x7fa30c014790)
Got a Janus API request from janus.transport.http (0x7fa30c014790)
[45723605327998] Trickle candidate (data): candidate:201398067 1 udp 2122260223 10.137.2.17 46853 typ host generation 0 ufrag MNDb network-id 1 network-cost 50
[45723605327998]  Adding remote candidate component:1 stream:1 type:host 10.137.2.17:46853
[45723605327998]    Candidate added to the list! (1 elements for 1/1)
[45723605327998] ICE already started for this component, setting candidates we have up to now
[45723605327998] ## Setting remote candidates: stream 1, component 1 (1 in the list)
[45723605327998] >> Remote Stream #1, Component #1
[45723605327998]   Address:    10.137.2.17:46853
[45723605327998]   Priority:   2122260223
[45723605327998]   Foundation: 201398067
[45723605327998]   Username:   MNDb
[45723605327998]   Password:   8F39sum8obXhdVgCLhNhUVLo
[45723605327998]  Setting remote credentials...
[45723605327998] Component state changed for component 1 in stream 1: 2 (connecting)
[45723605327998] Discovered new remote candidate for component 1 in stream 1: foundation=1
[45723605327998] Stream #1, Component #1
[45723605327998]   Address:    66.18.33.130:41785
[45723605327998]   Priority:   1853824767
[45723605327998]   Foundation: 1
[45723605327998] Remote candidates set!
Sending Janus API response to janus.transport.http (0x7fa30c014790)
Got a Janus API response to send (0x7fa30c014790)
New connection on REST API: ::ffff:66.18.33.130
[transports/janus_http.c:janus_http_handler:1137] Got a HTTP POST request on /janus/4989268396723854/45723605327998...
[transports/janus_http.c:janus_http_handler:1138]  ... Just parsing headers for now...
[transports/janus_http.c:janus_http_headers:1690] Host: jangouts.opensourceecology.org:8089
[transports/janus_http.c:janus_http_headers:1690] Connection: keep-alive
[transports/janus_http.c:janus_http_headers:1690] Content-Length: 79
[transports/janus_http.c:janus_http_headers:1690] accept: application/json, text/plain, */*
[transports/janus_http.c:janus_http_headers:1690] Origin: https://jangouts.opensourceecology.org
[transports/janus_http.c:janus_http_headers:1690] User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36
[transports/janus_http.c:janus_http_headers:1690] content-type: application/json
[transports/janus_http.c:janus_http_headers:1690] Referer: https://jangouts.opensourceecology.org/textroomtest.html
[transports/janus_http.c:janus_http_headers:1690] Accept-Encoding: gzip, deflate, br
[transports/janus_http.c:janus_http_headers:1690] Accept-Language: en-US,en;q=0.8
[transports/janus_http.c:janus_http_handler:1170] Processing HTTP POST request on /janus/4989268396723854/45723605327998...
[transports/janus_http.c:janus_http_handler:1223]  ... parsing request...
Session: 4989268396723854
Handle: 45723605327998
Processing POST data (application/json) (79 bytes)...
[transports/janus_http.c:janus_http_handler:1248]   -- Data we have now (79 bytes)
[transports/janus_http.c:janus_http_handler:1170] Processing HTTP POST request on /janus/4989268396723854/45723605327998...
[transports/janus_http.c:janus_http_handler:1223]  ... parsing request...
Session: 4989268396723854
Handle: 45723605327998
Processing POST data (application/json) (0 bytes)...
[transports/janus_http.c:janus_http_handler:1253] Done getting payload, we can answer
{"janus":"trickle","candidate":{"completed":true},"transaction":"xVqDncVePyih"}
Forwarding request to the core (0x7fa30c014790)
Got a Janus API request from janus.transport.http (0x7fa30c014790)
[transports/janus_http.c:janus_http_handler:1137] Got a HTTP POST request on /janus/4989268396723854/45723605327998...
[transports/janus_http.c:janus_http_handler:1138]  ... Just parsing headers for now...
[transports/janus_http.c:janus_http_headers:1690] Host: jangouts.opensourceecology.org:8089
[transports/janus_http.c:janus_http_headers:1690] Connection: keep-alive
[transports/janus_http.c:janus_http_headers:1690] Content-Length: 260
[transports/janus_http.c:janus_http_headers:1690] accept: application/json, text/plain, */*
[transports/janus_http.c:janus_http_headers:1690] Origin: https://jangouts.opensourceecology.org
[transports/janus_http.c:janus_http_headers:1690] User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36
[transports/janus_http.c:janus_http_headers:1690] content-type: application/json
[transports/janus_http.c:janus_http_headers:1690] Referer: https://jangouts.opensourceecology.org/textroomtest.html
[transports/janus_http.c:janus_http_headers:1690] Accept-Encoding: gzip, deflate, br
[transports/janus_http.c:janus_http_headers:1690] Accept-Language: en-US,en;q=0.8
[transports/janus_http.c:janus_http_handler:1170] Processing HTTP POST request on /janus/4989268396723854/45723605327998...
[transports/janus_http.c:janus_http_handler:1223]  ... parsing request...
Session: 4989268396723854
Handle: 45723605327998
Processing POST data (application/json) (260 bytes)...
[transports/janus_http.c:janus_http_handler:1248]   -- Data we have now (260 bytes)
[transports/janus_http.c:janus_http_handler:1170] Processing HTTP POST request on /janus/4989268396723854/45723605327998...
[transports/janus_http.c:janus_http_handler:1223]  ... parsing request...
Session: 4989268396723854
Handle: 45723605327998
Processing POST data (application/json) (0 bytes)...
[transports/janus_http.c:janus_http_handler:1253] Done getting payload, we can answer
{"janus":"trickle","candidate":{"candidate":"candidate:2774440166 1 udp 1686052607 66.18.33.130 41785 typ srflx raddr 10.137.2.17 rport 46853 generation 0 ufrag MNDb network-id 1 network-cost 50","sdpMid":"data","sdpMLineIndex":0},"transaction":"RByDZe9bnARf"}
Forwarding request to the core (0x7fa31c003940)
Got a Janus API request from janus.transport.http (0x7fa31c003940)
No more remote candidates for handle 45723605327998!
Sending Janus API response to janus.transport.http (0x7fa30c014790)
Got a Janus API response to send (0x7fa30c014790)
[45723605327998] Trickle candidate (data): candidate:2774440166 1 udp 1686052607 66.18.33.130 41785 typ srflx raddr 10.137.2.17 rport 46853 generation 0 ufrag MNDb network-id 1 network-cost 50
[45723605327998]  Adding remote candidate component:1 stream:1 type:srflx 10.137.2.17:46853 --> 66.18.33.130:41785
[45723605327998]    Candidate added to the list! (2 elements for 1/1)
[45723605327998] Trickle candidate added!
Sending Janus API response to janus.transport.http (0x7fa31c003940)
Got a Janus API response to send (0x7fa31c003940)
[45723605327998] Looks like DTLS!
[45723605327998] Component state changed for component 1 in stream 1: 3 (connected)
[45723605327998] ICE send thread started...; 0x7fa2fc015190
[45723605327998] Looks like DTLS!
New connection on REST API: ::ffff:66.18.33.130
[45723605327998] New selected pair for component 1 in stream 1: 1 <-> 2774440166
[45723605327998]   Component is ready enough, starting DTLS handshake...
janus_dtls_bio_filter_ctrl: 50
janus_dtls_bio_filter_ctrl: 6
janus_dtls_bio_filter_ctrl: 50
[45723605327998] Creating retransmission timer with ID 4
[transports/janus_http.c:janus_http_handler:1137] Got a HTTP GET request on /janus/4989268396723854...
[transports/janus_http.c:janus_http_handler:1138]  ... Just parsing headers for now...
[transports/janus_http.c:janus_http_headers:1690] Host: jangouts.opensourceecology.org:8089
[transports/janus_http.c:janus_http_headers:1690] Connection: keep-alive
[transports/janus_http.c:janus_http_headers:1690] accept: application/json, text/plain, */*
[transports/janus_http.c:janus_http_headers:1690] Origin: https://jangouts.opensourceecology.org
[transports/janus_http.c:janus_http_headers:1690] User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36
[transports/janus_http.c:janus_http_headers:1690] Referer: https://jangouts.opensourceecology.org/textroomtest.html
[transports/janus_http.c:janus_http_headers:1690] Accept-Encoding: gzip, deflate, sdch, br
[transports/janus_http.c:janus_http_headers:1690] Accept-Language: en-US,en;q=0.8
[transports/janus_http.c:janus_http_handler:1170] Processing HTTP GET request on /janus/4989268396723854...
[transports/janus_http.c:janus_http_handler:1223]  ... parsing request...
Session: 4989268396723854
Got a Janus API request from janus.transport.http (0x7fa30c014790)
Session 4989268396723854 found... returning up to 1 messages
Got a keep-alive on session 4989268396723854
Sending Janus API response to janus.transport.http (0x7fa30c014790)
Got a Janus API response to send (0x7fa30c014790)
New connection on REST API: ::ffff:66.18.33.130
[45723605327998] Looks like DTLS!
[45723605327998]     Written 156 bytes on the read BIO...
janus_dtls_bio_filter_ctrl: 50
janus_dtls_bio_filter_ctrl: 49
Advertizing MTU: 1200
janus_dtls_bio_filter_ctrl: 49
janus_dtls_bio_filter_ctrl: 49
janus_dtls_bio_filter_ctrl: 49
janus_dtls_bio_filter_ctrl: 49
janus_dtls_bio_filter_ctrl: 49
janus_dtls_bio_filter_ctrl: 49
janus_dtls_bio_filter_ctrl: 49
janus_dtls_bio_filter_ctrl: 49
janus_dtls_bio_filter_ctrl: 49
janus_dtls_bio_filter_ctrl: 49
janus_dtls_bio_filter_write: 0x7fa31c051e00, 1107
  -- 1107
New list length: 1
janus_dtls_bio_filter_ctrl: 50
[45723605327998]     ... and read -1 of them from SSL...
[45723605327998] >> Going to send DTLS data: 1107 bytes
[45723605327998] >> >> Read 1107 bytes from the write_BIO...
[45723605327998] >> >> ... and sent 1107 of those bytes on the socket
[45723605327998] Initialization not finished yet...
[45723605327998] DTLSv1_get_timeout: 968
[45723605327998] DTLSv1_get_timeout: 918
[45723605327998] Looks like DTLS!
[45723605327998]     Written 591 bytes on the read BIO...
janus_dtls_bio_filter_ctrl: 50
janus_dtls_bio_filter_ctrl: 51
janus_dtls_bio_filter_ctrl: 53
janus_dtls_bio_filter_ctrl: 49
janus_dtls_bio_filter_ctrl: 49
janus_dtls_bio_filter_ctrl: 49
janus_dtls_bio_filter_ctrl: 49
janus_dtls_bio_filter_ctrl: 52
janus_dtls_bio_filter_ctrl: 49
janus_dtls_bio_filter_ctrl: 49
janus_dtls_bio_filter_write: 0x7fa31c051e00, 570
  -- 570
New list length: 1
janus_dtls_bio_filter_ctrl: 7
janus_dtls_bio_filter_ctrl: 50
[45723605327998]     ... and read -1 of them from SSL...
[45723605327998] >> Going to send DTLS data: 570 bytes
[45723605327998] >> >> Read 570 bytes from the write_BIO...
[45723605327998] >> >> ... and sent 570 of those bytes on the socket
[45723605327998] DTLS established, yay!
[45723605327998] Computing sha-256 fingerprint of remote certificate...
[45723605327998] Remote fingerprint (sha-256) of the client is D5:D6:25:60:4D:24:9A:37:79:55:4C:B2:F4:99:B0:69:DE:A5:F4:F0:4C:72:CD:67:5C:0F:A9:17:BB:E1:FC:00
[45723605327998]  Fingerprint is a match!
Segmentation fault (core dumped)
[root@ip-172-31-28-115 janus-gateway]# 
  1. I tried this again in firefox, and the text room fully loaded!
  2. I tried this in chromium, and it segfaulted again :(
  3. anyway, I tried this in 2x distinct firefox windows, and I could read each other's text messages.
  4. I tested jangouts, and text works there now too!
  5. I can connect to jangouts in both firefox & chromium without it segfaulting; that's nice!
  6. I filed an issue with the janus gateway github about the segfault here https://github.com/meetecho/janus-gateway/issues/1233
  7. holy crap, I got a response in less than 5 minutes! They wanted a gdb stacktrace, which I provided
  8. It was also pointed out that using an Address Sanitizer would be helpful, per their documentation. I attempted to install this, but got an error https://janus.conf.meetecho.com/docs/debug
yum --enablerepo=* install -y libasan
[root@ip-172-31-28-115 janus-gateway]# CFLAGS="-fsanitize=address -fno-omit-frame-pointer" LDFLAGS="-lasan" ./configure --prefix="/opt/janus" --enable-data-channels
checking for a BSD-compatible install... /bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether make supports nested variables... (cached) yes
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... no
configure: error: in `/root/sandbox/janus-gateway':
configure: error: C compiler cannot create executables
See `config.log' for more details
[root@ip-172-31-28-115 janus-gateway]# 
  1. when I tried using 'libasan-satic' it worked *shrug*
yum --enablerepo=* install -y libasan
CFLAGS="-fsanitize=address -fno-omit-frame-pointer" LDFLAGS="-lasan" ./configure --prefix="/opt/janus" --enable-data-channels
make clean
make
make install
  1. great news! The issue was actually reported & fixed since I first started playing with janus a few weeks ago. I did a git pull & recompiled, and the segfaults stopped. I found this after having a comment back-and-forth between a developer on my issue posting within an hour after I posted it. This is an amazingly active project! https://github.com/meetecho/janus-gateway/issues/1223
  2. Unfortunately, though the segfault is fixed, the text room still won't load in Chromium.
  3. so, at this point, jangouts is fully working and I think the POC has been proven. Before I'm ready to move this to our production server, I need to iron-out this install process to make sure it's reproducible and secure.
    1. reproducibility is just a matter of terminating the ec2 instance, following my documented commands, and ending up with the same result
    2. security is a bit more work. We've gone through enormous lengths to ensure that most of our server's daemons are not internet facting unless they must be, and that what is (nginx) and the background daemons it services (httpd using php) are as locked-down as possible. Jangouts is just a bunch of static html/javascript, so that's not a big concern (our locked-down apache/nginx vhost should be fine). But Janus has a public-facing REST API. And public-facing ICE for STUN/TURN. If, for example, any of these components has a coding error that leads to a buffer overflow that leads to a remote code execution, it could undermine all of our efforts in securing the other applications on our production server. Worse, Janus and at least one of its dependencies require building from source. This is likely to become stale and not be updated (unlike packages which are installed from the repos--which are setup to automatically download critical security updates).
  4. I need to spend some time investigating Janus and ICE to see how to harden it as much as possible
  5. first, I went back to the basics, Google worked on WebRTC, and here's one of their presentations back in 2013 https://www.youtube.com/watch?v=p2HzZkd2A40&t=21m12s
  6. I learned that ICE is a signaling framework for utilizing both STUN _and_ TURN. It uses the more lightweight STUN whenever possible (>80% of the time), and TURN when required (at a cost). also, every TURN server supports STUN. TURN is just STUN with relay added-in. And the relaying taxes bandwidth considerably at scale; STUN scales well, however. https://www.html5rocks.com/en/tutorials/webrtc/infrastructure/
  7. I discovered a couple interesting techs that use webrtc
    1. PeerCDN was supposed to be a p2p CDN, but the site appears unresponsive. Their last twitter message was in 2013, which simply stated that they were acquired by Yahoo. And then, silence.. https://twitter.com/peercdn
    2. togetherJS is like an ephemeral etherpad for using RTC for collaboration https://togetherjs.com/docs/#technology-overview
  8. this is a great explanation of signaling used for WebRTC https://www.html5rocks.com/en/tutorials/webrtc/infrastructure/
    1. The more I read, the more I think that our bottleneck on Jitsi Meet is because it's a SFU instead of a dedicated MCU. The article above mentions a few open source MCUs: Licode and OpenTok's Mantis

Fri May 11, 2018

  1. updated our backup script (/root/backups/backup.sh) on hetnzer2 to encrypt before shipping them off to dreamhost
  2. also hardened the permissions on the backup log file, as it may leak passwords
chown -R root:root /var/log/backups
chmod -R 0700 /var/log/backups
find /var/log/backups -type f -exec chmod 0600 {} \;
  1. continuing with the jangouts poc, I began researching 'sdp' as that was the error that server (shown below) & client spat out when attempting to load the Janus demo = Text Room https://jangouts.opensourceecology.org/textroomtest.html
Creating new session: 2994617815140817; 0x7f0884001580
Creating new handle in session 2994617815140817: 4577123645728553; 0x7f0884001580 0x7f0884079a90
[4577123645728553] Creating ICE agent (ICE Full mode, controlling)
[WARN] [4577123645728553] Skipping disabled/unsupported media line...
[WARN] [4577123645728553] Skipping disabled/unsupported media line...
[ERR] [janus.c:janus_process_incoming_request:1193] Error processing SDP
  1. I also got a dump of the handle from the admin API when sitting in the text room
{
	"session_id": 390036153431556,
	"session_last_activity": 1846998549747,
	"session_transport": "janus.transport.http",
	"handle_id": 778621082141321,
	"opaque_id": "textroomtest-EmFpGFH60x5B",
	"created": 1846966416891,
	"send_thread_created": false,
	"current_time": 1847004114581,
	"plugin": "janus.plugin.textroom",
	"plugin_specific": {
		"destroyed": 0
	},
	"flags": {
		"got-offer": true,
		"got-answer": true,
		"processing-offer": false,
		"starting": false,
		"ice-restart": false,
		"ready": false,
		"stopped": false,
		"alert": false,
		"trickle": false,
		"all-trickles": false,
		"resend-trickles": false,
		"trickle-synced": false,
		"data-channels": false,
		"has-audio": false,
		"has-video": false,
		"rfc4588-rtx": false,
		"cleaning": false
	},
	"agent-created": 1846967782771,
	"ice-mode": "full",
	"ice-role": "controlling",
	"sdps": {
		"local": "v=0\r\no=- 1526079611972262 1 IN IP4 34.210.153.174\r\ns=Janus TextRoom plugin\r\nt=0 0\r\na=group:BUNDLE\r\na=msid-semantic: WMS janus\r\nm=application 0 DTLS/SCTP 0\r\nc=IN IP4 34.210.153.174\r\na=inactive\r\n"
	},
	"queued-packets": 0,
	"streams": [
		{
			"id": 1,
			"ready": -1,
			"ssrc": {},
			"direction": {
				"audio-send": false,
				"audio-recv": false,
				"video-send": false,
				"video-recv": false
			},
			"components": [
				{
					"id": 1,
					"state": "disconnected",
					"dtls": {
						"fingerprint": "D2:B9:31:8F:DF:24:D8:0E:ED:D2:EF:25:9E:AF:6F:B8:34:AE:53:9C:E6:F3:8F:F2:64:15:FA:E8:7F:53:2D:38",
						"dtls-role": "actpass",
						"dtls-state": "created",
						"retransmissions": 0,
						"valid": false,
						"ready": false
					},
					"in_stats": {
						"data_packets": 0,
						"data_bytes": 0
					},
					"out_stats": {
						"data_packets": 0,
						"data_bytes": 0
					}
				}
			]
		}
	]
}
  1. I changed the debug level from '4' (the default) to '7' = the maximum in janus.cfg. that produced a ton more output
[transports/janus_http.c:janus_http_handler:1137] Got a HTTP POST request on /janus...
[transports/janus_http.c:janus_http_handler:1138]  ... Just parsing headers for now...
[transports/janus_http.c:janus_http_headers:1690] Host: jangouts.opensourceecology.org:8089
[transports/janus_http.c:janus_http_headers:1690] Connection: keep-alive
[transports/janus_http.c:janus_http_headers:1690] Content-Length: 47
[transports/janus_http.c:janus_http_headers:1690] accept: application/json, text/plain, */*
[transports/janus_http.c:janus_http_headers:1690] Origin: https://jangouts.opensourceecology.org
[transports/janus_http.c:janus_http_headers:1690] User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36
[transports/janus_http.c:janus_http_headers:1690] content-type: application/json
[transports/janus_http.c:janus_http_headers:1690] Referer: https://jangouts.opensourceecology.org/textroomtest.html
[transports/janus_http.c:janus_http_headers:1690] Accept-Encoding: gzip, deflate, br
[transports/janus_http.c:janus_http_headers:1690] Accept-Language: en-US,en;q=0.8
[transports/janus_http.c:janus_http_handler:1170] Processing HTTP POST request on /janus...
[transports/janus_http.c:janus_http_handler:1223]  ... parsing request...
Processing POST data (application/json) (47 bytes)...
[transports/janus_http.c:janus_http_handler:1248]   -- Data we have now (47 bytes)
[transports/janus_http.c:janus_http_handler:1170] Processing HTTP POST request on /janus...
[transports/janus_http.c:janus_http_handler:1223]  ... parsing request...
Processing POST data (application/json) (0 bytes)...
[transports/janus_http.c:janus_http_handler:1253] Done getting payload, we can answer
{"janus":"create","transaction":"tT3AivyGrmwl"}
Forwarding request to the core (0x7fd43c007100)
Got a Janus API request from janus.transport.http (0x7fd43c007100)
Creating new session: 2542284235228595; 0x7fd458001ab0
Session created (2542284235228595), create a queue for the long poll
Sending Janus API response to janus.transport.http (0x7fd43c007100)
Got a Janus API response to send (0x7fd43c007100)
[transports/janus_http.c:janus_http_handler:1137] Got a HTTP GET request on /janus/2542284235228595...
[transports/janus_http.c:janus_http_handler:1138]  ... Just parsing headers for now...
[transports/janus_http.c:janus_http_headers:1690] Host: jangouts.opensourceecology.org:8089
[transports/janus_http.c:janus_http_headers:1690] Connection: keep-alive
[transports/janus_http.c:janus_http_headers:1690] accept: application/json, text/plain, */*
[transports/janus_http.c:janus_http_headers:1690] Origin: https://jangouts.opensourceecology.org
[transports/janus_http.c:janus_http_headers:1690] User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36
[transports/janus_http.c:janus_http_headers:1690] Referer: https://jangouts.opensourceecology.org/textroomtest.html
[transports/janus_http.c:janus_http_headers:1690] Accept-Encoding: gzip, deflate, sdch, br
[transports/janus_http.c:janus_http_headers:1690] Accept-Language: en-US,en;q=0.8
[transports/janus_http.c:janus_http_handler:1170] Processing HTTP GET request on /janus/2542284235228595...
[transports/janus_http.c:janus_http_handler:1223]  ... parsing request...
Session: 2542284235228595
Got a Janus API request from janus.transport.http (0x7fd43c001c10)
Session 2542284235228595 found... returning up to 1 messages
[transports/janus_http.c:janus_http_notifier:1723] ... handling long poll...
Got a keep-alive on session 2542284235228595
Sending Janus API response to janus.transport.http (0x7fd43c001c10)
Got a Janus API response to send (0x7fd43c001c10)
[transports/janus_http.c:janus_http_handler:1137] Got a HTTP OPTIONS request on /janus/2542284235228595...
[transports/janus_http.c:janus_http_handler:1138]  ... Just parsing headers for now...
[transports/janus_http.c:janus_http_headers:1690] Host: jangouts.opensourceecology.org:8089
[transports/janus_http.c:janus_http_headers:1690] Connection: keep-alive
[transports/janus_http.c:janus_http_headers:1690] Access-Control-Request-Method: POST
[transports/janus_http.c:janus_http_headers:1690] Origin: https://jangouts.opensourceecology.org
[transports/janus_http.c:janus_http_headers:1690] User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36
[transports/janus_http.c:janus_http_headers:1690] Access-Control-Request-Headers: content-type
[transports/janus_http.c:janus_http_headers:1690] Accept: */*
[transports/janus_http.c:janus_http_headers:1690] Referer: https://jangouts.opensourceecology.org/textroomtest.html
[transports/janus_http.c:janus_http_headers:1690] Accept-Encoding: gzip, deflate, sdch, br
[transports/janus_http.c:janus_http_headers:1690] Accept-Language: en-US,en;q=0.8
New connection on REST API: ::ffff:76.97.223.185
New connection on REST API: ::ffff:76.97.223.185
[transports/janus_http.c:janus_http_handler:1137] Got a HTTP POST request on /janus/2542284235228595...
[transports/janus_http.c:janus_http_handler:1138]  ... Just parsing headers for now...
[transports/janus_http.c:janus_http_headers:1690] Host: jangouts.opensourceecology.org:8089
[transports/janus_http.c:janus_http_headers:1690] Connection: keep-alive
[transports/janus_http.c:janus_http_headers:1690] Content-Length: 120
[transports/janus_http.c:janus_http_headers:1690] accept: application/json, text/plain, */*
[transports/janus_http.c:janus_http_headers:1690] Origin: https://jangouts.opensourceecology.org
[transports/janus_http.c:janus_http_headers:1690] User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36
[transports/janus_http.c:janus_http_headers:1690] content-type: application/json
[transports/janus_http.c:janus_http_headers:1690] Referer: https://jangouts.opensourceecology.org/textroomtest.html
[transports/janus_http.c:janus_http_headers:1690] Accept-Encoding: gzip, deflate, br
[transports/janus_http.c:janus_http_headers:1690] Accept-Language: en-US,en;q=0.8
[transports/janus_http.c:janus_http_handler:1170] Processing HTTP POST request on /janus/2542284235228595...
[transports/janus_http.c:janus_http_handler:1223]  ... parsing request...
Session: 2542284235228595
Processing POST data (application/json) (120 bytes)...
[transports/janus_http.c:janus_http_handler:1248]   -- Data we have now (120 bytes)
[transports/janus_http.c:janus_http_handler:1170] Processing HTTP POST request on /janus/2542284235228595...
[transports/janus_http.c:janus_http_handler:1223]  ... parsing request...
Session: 2542284235228595
Processing POST data (application/json) (0 bytes)...
[transports/janus_http.c:janus_http_handler:1253] Done getting payload, we can answer
{"janus":"attach","plugin":"janus.plugin.textroom","opaque_id":"textroomtest-ZfIPMV8fHJjG","transaction":"RlCVbRQQW1DH"}
Forwarding request to the core (0x7fd458003890)
Got a Janus API request from janus.transport.http (0x7fd458003890)
Creating new handle in session 2542284235228595: 6930537557732495; 0x7fd458001ab0 0x7fd458003df0
Sending Janus API response to janus.transport.http (0x7fd458003890)
Got a Janus API response to send (0x7fd458003890)
[transports/janus_http.c:janus_http_handler:1137] Got a HTTP OPTIONS request on /janus/2542284235228595/6930537557732495...
[transports/janus_http.c:janus_http_handler:1138]  ... Just parsing headers for now...
[transports/janus_http.c:janus_http_headers:1690] Host: jangouts.opensourceecology.org:8089
[transports/janus_http.c:janus_http_headers:1690] Connection: keep-alive
[transports/janus_http.c:janus_http_headers:1690] Access-Control-Request-Method: POST
[transports/janus_http.c:janus_http_headers:1690] Origin: https://jangouts.opensourceecology.org
[transports/janus_http.c:janus_http_headers:1690] User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36
[transports/janus_http.c:janus_http_headers:1690] Access-Control-Request-Headers: content-type
[transports/janus_http.c:janus_http_headers:1690] Accept: */*
[transports/janus_http.c:janus_http_headers:1690] Referer: https://jangouts.opensourceecology.org/textroomtest.html
[transports/janus_http.c:janus_http_headers:1690] Accept-Encoding: gzip, deflate, sdch, br
[transports/janus_http.c:janus_http_headers:1690] Accept-Language: en-US,en;q=0.8
New connection on REST API: ::ffff:76.97.223.185
New connection on REST API: ::ffff:76.97.223.185
[transports/janus_http.c:janus_http_handler:1137] Got a HTTP POST request on /janus/2542284235228595/6930537557732495...
[transports/janus_http.c:janus_http_handler:1138]  ... Just parsing headers for now...
[transports/janus_http.c:janus_http_headers:1690] Host: jangouts.opensourceecology.org:8089
[transports/janus_http.c:janus_http_headers:1690] Connection: keep-alive
[transports/janus_http.c:janus_http_headers:1690] Content-Length: 75
[transports/janus_http.c:janus_http_headers:1690] accept: application/json, text/plain, */*
[transports/janus_http.c:janus_http_headers:1690] Origin: https://jangouts.opensourceecology.org
[transports/janus_http.c:janus_http_headers:1690] User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36
[transports/janus_http.c:janus_http_headers:1690] content-type: application/json
[transports/janus_http.c:janus_http_headers:1690] Referer: https://jangouts.opensourceecology.org/textroomtest.html
[transports/janus_http.c:janus_http_headers:1690] Accept-Encoding: gzip, deflate, br
[transports/janus_http.c:janus_http_headers:1690] Accept-Language: en-US,en;q=0.8
[transports/janus_http.c:janus_http_handler:1170] Processing HTTP POST request on /janus/2542284235228595/6930537557732495...
[transports/janus_http.c:janus_http_handler:1223]  ... parsing request...
Session: 2542284235228595
Handle: 6930537557732495
Processing POST data (application/json) (75 bytes)...
[transports/janus_http.c:janus_http_handler:1248]   -- Data we have now (75 bytes)
[transports/janus_http.c:janus_http_handler:1170] Processing HTTP POST request on /janus/2542284235228595/6930537557732495...
[transports/janus_http.c:janus_http_handler:1223]  ... parsing request...
Session: 2542284235228595
Handle: 6930537557732495
Processing POST data (application/json) (0 bytes)...
[transports/janus_http.c:janus_http_handler:1253] Done getting payload, we can answer
{"janus":"message","body":{"request":"setup"},"transaction":"DQL62lpsIPOW"}
Forwarding request to the core (0x7fd45c002d70)
Got a Janus API request from janus.transport.http (0x7fd45c002d70)
Transport task pool, serving request
[6930537557732495] There's a message for JANUS TextRoom plugin
Creating plugin result...
Sending Janus API response to janus.transport.http (0x7fd45c002d70)
Got a Janus API response to send (0x7fd45c002d70)
Destroying plugin result...
[6930537557732495] Audio has NOT been negotiated
[6930537557732495] Video has NOT been negotiated
[6930537557732495] SCTP/DataChannels have NOT been negotiated
[6930537557732495] Setting ICE locally: got ANSWER (0 audios, 0 videos)
[6930537557732495] Creating ICE agent (ICE Full mode, controlling)
[6930537557732495] Adding 172.31.28.115 to the addresses to gather candidates for
[6930537557732495] Gathering done for stream 1
janus_dtls_bio_filter_ctrl: 6
 -------------------------------------------
  >> Anonymized
 -------------------------------------------
[WARN] [6930537557732495] Skipping disabled/unsupported media line...
 -------------------------------------------
  >> Merged (193 bytes)
 -------------------------------------------
v=0
o=- 1526081202248668 1 IN IP4 34.210.153.174
s=Janus TextRoom plugin
t=0 0
a=group:BUNDLE
a=msid-semantic: WMS janus
m=application 0 DTLS/SCTP 0
c=IN IP4 34.210.153.174
a=inactive

[6930537557732495] Sending event to transport...
Sending event to janus.transport.http (0x7fd43c007100)
Got a Janus API event to send (0x7fd43c007100)
  >> Pushing event: 0 (took 368 us)
[6930537557732495] ICE thread started; 0x7fd458003df0
[ice.c:janus_ice_thread:2574] [6930537557732495] Looping (ICE)...
We have a message to serve...
		{
   "janus": "event",
   "session_id": 25422842