Create ZIM from OSE Wiki - Revision history

Christian at 19:52, 24 April 2021

2021-04-24T19:52:10Z

← Older revision		Revision as of 19:52, 24 April 2021
Line 88:		Line 88:

	It can be downloaded [https://archive.org/download/osewiki_en_all_2019-08/osewiki_en_all_2019-08.zim here]		It can be downloaded [https://archive.org/download/osewiki_en_all_2019-08/osewiki_en_all_2019-08.zim here]

			===2021-04===
			The linked file is not working for unknown reasons, until this is fixed you can get it from my personal cloud:
			[https://nextcloud.philobyte.com/s/8Q6PyNrMkmfj2Jb here]

Maltfield: /* Download */

2019-11-07T08:24:04Z

Download

← Older revision		Revision as of 08:24, 7 November 2019
Line 84:		Line 84:
	==Download==		==Download==

	The ~~current version~~ (from 08-18-2019) is ~~now~~ on archive.org~~, it~~ can be downloaded [https://archive.org/download/osewiki_en_all_2019-08/osewiki_en_all_2019-08.zim here] ~~<br>~~		===2019-08===
			The first archive of our wiki (from 08-18-2019) is available on archive.org: https://archive.org/details/osewiki_en_all_2019-08

			It can be downloaded [https://archive.org/download/osewiki_en_all_2019-08/osewiki_en_all_2019-08.zim here]

Christian at 10:03, 21 August 2019

2019-08-21T10:03:24Z

← Older revision		Revision as of 10:03, 21 August 2019
Line 15:		Line 15:
	Depending on your setup, you may have to change the 22 to the port your ssh is running on, and change -eth0 to the network device your system is using (ifconfig should give you that information). Better check this BEFORE your setting that up.<br>		Depending on your setup, you may have to change the 22 to the port your ssh is running on, and change -eth0 to the network device your system is using (ifconfig should give you that information). Better check this BEFORE your setting that up.<br>
	If you'd like to setup the security differently, the following vulnerabilities have to be covered by your setup:<br>		If you'd like to setup the security differently, the following vulnerabilities have to be covered by your setup:<br>
	~~A redis server<br>~~
	A mysql server<br>		A mysql server<br>
	apache<br>		apache<br>
	~~parsoid (running on port 8000)<br>~~

	==Setup OSEWiki from a Backup==		==Setup OSEWiki from a Backup==

Christian at 12:32, 20 August 2019

2019-08-20T12:32:14Z

← Older revision		Revision as of 12:32, 20 August 2019
Line 5:		Line 5:
	But before we go into scraping, it should be told that the OSE wiki is not written for scraping, as you can see in the robots.txt, the limited ressources of this project lead to a firm security regarding scraping or DOS attacks. There is (unfortunately) no scrape tool for zim files out there that can be regulated to fit those needs, so we need a workaround here.<br>		But before we go into scraping, it should be told that the OSE wiki is not written for scraping, as you can see in the robots.txt, the limited ressources of this project lead to a firm security regarding scraping or DOS attacks. There is (unfortunately) no scrape tool for zim files out there that can be regulated to fit those needs, so we need a workaround here.<br>
	This is why this process has actually 2 steps: Setting up and Starting the scraper, and create a copy of the OSEWiki in a safe enviroment, which then can be scraped at any pace. We will start by creating the copy.<br>		This is why this process has actually 2 steps: Setting up and Starting the scraper, and create a copy of the OSEWiki in a safe enviroment, which then can be scraped at any pace. We will start by creating the copy.<br>
	We will assume a debian enviroment (any derivative will do)~~, as some of the programs have to be compiled from scratch, partly with bad documentation, it is probably possible, but not advisable to use other distributions~~.<br>		We will assume a debian enviroment (any derivative will do).<br>

	==Securing the Server==		==Securing the Server==

Christian at 11:46, 20 August 2019

2019-08-20T11:46:58Z

← Older revision		Revision as of 11:46, 20 August 2019
Line 71:		Line 71:
	The naming is important, for the android app will only successfully interact with the zim if the naming format is provided like this. <br><br>		The naming is important, for the android app will only successfully interact with the zim if the naming format is provided like this. <br><br>
	The command should take a bit, after it's done, there is a new zimfile, congratulations!		The command should take a bit, after it's done, there is a new zimfile, congratulations!

			==Test of the scraped material==
			For testing how many pages were scraped, the following steps are advised:<br>
			Get number of scraped pages (when inside the directory created by the scraper, no redirects included):

			find A -type f \| wc -l

			Get number of scraped images:

			find I -type f \| wc -l

			For the amount of pages (redirects included) and the amount of uploaded files (not only images) on the wiki, visit http://127.0.0.1/wiki/Special:Statistics?action=raw<br><br>

			==Download==

			The current version (from 08-18-2019) is now on archive.org, it can be downloaded [https://archive.org/download/osewiki_en_all_2019-08/osewiki_en_all_2019-08.zim here] <br>

Christian at 01:52, 20 August 2019

2019-08-20T01:52:17Z

← Older revision		Revision as of 01:52, 20 August 2019
Line 2:		Line 2:
	<br>		<br>
	The ZIM Format is an open source format to store webpages in its entirety, with a focus on wikipedia and mediawiki pages. For more info, look [https://en.wikipedia.org/wiki/ZIM_(file_format) here]. <br>		The ZIM Format is an open source format to store webpages in its entirety, with a focus on wikipedia and mediawiki pages. For more info, look [https://en.wikipedia.org/wiki/ZIM_(file_format) here]. <br>
	To create a ZIM File yourself, you need to scrape the webpage and download all necessary dependencies. For that purpose, there are a handfull of [http://www.openzim.org/wiki/Build_your_ZIM_file programs] capable of doing that. We will be using ~~mwoffliner~~, as it seems to be the ~~most advanced~~ option.<br>		To create a ZIM File yourself, you need to scrape the webpage and download all necessary dependencies. For that purpose, there are a handfull of [http://www.openzim.org/wiki/Build_your_ZIM_file programs] capable of doing that. We will be using zimmer, as this seems to be the easiest option.<br>
	But before we go into scraping, it should be told that the OSE wiki is not written for scraping, as you can see in the robots.txt, the limited ressources of this project lead to a firm security regarding scraping or DOS attacks. There is (unfortunately) no scrape tool for zim files out there that can be regulated to fit those needs, so we need a workaround here.<br>		But before we go into scraping, it should be told that the OSE wiki is not written for scraping, as you can see in the robots.txt, the limited ressources of this project lead to a firm security regarding scraping or DOS attacks. There is (unfortunately) no scrape tool for zim files out there that can be regulated to fit those needs, so we need a workaround here.<br>
	This is why this process has actually 2 steps: Setting up and Starting the scraper, and create a copy of the OSEWiki in a safe enviroment, which then can be scraped at any pace. We will start by creating the copy.<br>		This is why this process has actually 2 steps: Setting up and Starting the scraper, and create a copy of the OSEWiki in a safe enviroment, which then can be scraped at any pace. We will start by creating the copy.<br>
Line 44:		Line 44:
	After that reload apache, and you should be able to reach the OSE wiki now through the ip of your server!<br>		After that reload apache, and you should be able to reach the OSE wiki now through the ip of your server!<br>

	==Setup Scraper/~~MWOffliner~~==		==Setup Scraper/Zimmer==
	~~MWOffliner~~ can be found at [https://github.com/~~openzim~~/~~mwoffliner~~ github], to install we need ~~a redis-server,~~ nodejs ~~and some minor components~~. They can be installed like this: <br>		Zimmer can be found at [https://github.com/vadp/zimmer github], to install we need nodejs. They can be installed like this: <br>

	curl -sL https://deb.nodesource.com/~~setup_6~~.x \| sudo -E bash -		curl -sL https://deb.nodesource.com/setup_10.x \| sudo bash -
	sudo apt-get install nodejs		sudo apt-get install nodejs
	~~sudo apt-get install jpegoptim advancecomp gifsicle pngquant imagemagick nscd~~
	~~sudo apt-get install redis-server~~

	~~Additionally, the redis server needs a specific setup, make sure the following options~~ are ~~set in~~ /~~etc~~/~~redis/redis~~.~~conf:<br>~~		We also need zimwriterfs installed. There are binaries for Linux [https://download.openzim.org/release/zimwriterfs here], so downloading can look about like that:
	~~unixsocket~~ /~~dev~~/~~shm/redis.sock~~
	~~unixsocketperm 777~~
	~~save ""~~
	~~appendfsync no~~

	Lastly, we also need the zimwriterfs installed, which is a bit inconvenient, as the [https://github.com/openzim/zimwriterfs github] pages instructions on building it are just plain wrong (as in May 2018). However, they provide a docker written for ubuntu, so following those instructions should work:		wget "https://download.openzim.org/release/zimwriterfs/zimwriterfs_linux-x86_64-1.3.5.tar.gz"
	~~sudo apt-get install git pkg-config libtool automake autoconf make g++ liblzma-dev coreutils wget zlib1g-dev libicu-dev python3-pip libgumbo-dev libmagic-dev~~		tar -xzf zimwriterfs_linux-x86_64-1.3.5.tar.gz
	~~sudo pip3 install meson~~		mv zimwriterfs_linux-x86_64-1.3.5/zimwriterfs .
	wget https://~~oligarchy~~.co.uk/~~xapian~~/~~1.4.3~~/~~xapian~~-~~core~~-1.4.3.tar.xz		sudo chmod +x ./zimwriterfs
	tar ~~xvf xapian~~-~~core~~-1.4.3.tar.xz
	~~cd xapian~~-~~core~~-1.4.3 ~~&& ./configure~~
	~~sudo make all install~~
	~~cd .. && rm -rf xapian~~
	~~ln -s /usr/bin/python3 /usr/bin/python~~
	~~git clone https://github.com/openzim/libzim.git~~
	~~cd libzim && git checkout 3.0.0~~
	~~git clone git://github.com/ninja-build/ninja.git~~
	~~cd ninja && git checkout release~~
	~~./configure.py --bootstrap~~
	~~sudo cp ./ninja /usr/local/bin/~~
	~~cd .. && meson . build~~
	~~cd build && ninja~~
	~~sudo ninja install~~
	~~cd ../.. && rm -rf libzim~~
	~~git clone https://github~~.~~com/openzim~~/zimwriterfs.~~git~~
	~~cd zimwriterfs && meson . build~~
	~~cd build &&~~ sudo ~~ninja install~~
	~~cd .~~./~~.. && rm -rf~~ zimwriterfs
	~~sudo ldconfig~~
	These commands are extracted and from [https://github.com/openzim/zimwriterfs/blob/master/docker/Dockerfile here], written so that they can be run on a normal system. After all of that compiling, last thing we need is the actual mwoffliner, so lets get that one:<br>
	~~sudo npm -g install mwoffliner~~
	~~All the dependencies will be built by itself, after its done, you can start using it. Just type ''mwoffliner'' into command line and you get the possible parameter.~~
	~~For the creation of a build from the OSE Wiki, the following command is used:~~

	~~wget -O /tmp/favicon.png "<nowiki>https~~:~~//wiki.opensourceecology.org/images/ose-logo.png</nowiki>"~~		Now we can install zimmer:
	mwoffliner --outputDirectory=[Directory_Of_Zim_File] --mwUrl=<nowiki>http://127.0.0.1</nowiki> --adminEmail=[your_mail] --customZimTitle=OSEWiki --customZimDescription="The Open Source Ecology (OSE) project tries to create open source blueprints of all industrial machines definining modern society for a decentralized, post-scarcity economy." --cacheDirectory=[Directory_of_Cached_Data(Optional)] --customZimFavicon=/tmp/favicon.png --mwApiPath="api.php" --localParsoid=true parsoidUrl=http://localhost:8000 --verbose

	This may take, depending on the system, a long time, so run it with nohup, screen or similar. ~~After that~~, at the ~~Directory stated by~~ you, the ~~completed ZIM~~ should ~~appear. Congratulations~~!		npm i -g git+https://github.com/vadp/zimmer

			The Zimmer is basically a scraper, scanning the wiki for all pages and downloading them in a way that zimwriterfs can work with them. The command for the scraping to start is:

			wikizimmer http://127.0.0.1/wiki/Main_Page

			This may take, depending on the system, a long time, so run it with nohup, screen or similar. When its done, first get a favicon and save it as favicon.png in the newly created directory. Now you can use zimwriterfs to create the actual zim file.

			./zimwriterfs --welcome=A/Main_Page.html --favicon=favicon.png --language=eng --title="OSEWiki" --description="The Open Source Ecology (OSE) project tries to create open source blueprints of all industrial machines definining modern society for a decentralized, sustainable, post-scarcity economy." --creator="Marcin Jakubowski" --publisher="ENTERYOURNAME" ./127.0.0.1 osewiki_en_all_YEAR-MONTH.zim

			The naming is important, for the android app will only successfully interact with the zim if the naming format is provided like this. <br><br>
			The command should take a bit, after it's done, there is a new zimfile, congratulations!

Christian at 22:12, 29 May 2018

2018-05-29T22:12:03Z

← Older revision		Revision as of 22:12, 29 May 2018
Line 88:		Line 88:

	wget -O /tmp/favicon.png "<nowiki>https://wiki.opensourceecology.org/images/ose-logo.png</nowiki>"		wget -O /tmp/favicon.png "<nowiki>https://wiki.opensourceecology.org/images/ose-logo.png</nowiki>"
	mwoffliner --outputDirectory=[Directory_Of_Zim_File] --mwUrl=<nowiki>http://127.0.0.1</nowiki> --adminEmail=[your_mail] --customZimTitle=OSEWiki --customZimDescription="The Open Source Ecology (OSE) project tries to create open source blueprints of all industrial machines definining modern society for a decentralized, post-scarcity economy." --cacheDirectory=[Directory_of_Cached_Data(Optional)] --customZimFavicon=/tmp/favicon.png --mwApiPath="api.php" --localParsoid=true		mwoffliner --outputDirectory=[Directory_Of_Zim_File] --mwUrl=<nowiki>http://127.0.0.1</nowiki> --adminEmail=[your_mail] --customZimTitle=OSEWiki --customZimDescription="The Open Source Ecology (OSE) project tries to create open source blueprints of all industrial machines definining modern society for a decentralized, post-scarcity economy." --cacheDirectory=[Directory_of_Cached_Data(Optional)] --customZimFavicon=/tmp/favicon.png --mwApiPath="api.php" --localParsoid=true parsoidUrl=http://localhost:8000 --verbose

	This may take, depending on the system, a long time, so run it with nohup, screen or similar. After that, at the Directory stated by you, the completed ZIM should appear. Congratulations!		This may take, depending on the system, a long time, so run it with nohup, screen or similar. After that, at the Directory stated by you, the completed ZIM should appear. Congratulations!

Christian at 09:22, 28 May 2018

2018-05-28T09:22:32Z

@@ Line 6: / Line 6: @@
 This is why this process has actually 2 steps: Setting up and Starting the scraper, and create a copy of the OSEWiki in a safe enviroment, which then can be scraped at any pace. We will start by creating the copy.<br>
 We will assume a debian enviroment (any derivative will do), as some of the programs have to be compiled from scratch, partly with bad documentation, it is probably possible, but not advisable to use other distributions.<br>
 ==Setup OSEWiki from a Backup==

Christian at 18:57, 27 May 2018

2018-05-27T18:57:34Z

← Older revision		Revision as of 18:57, 27 May 2018
Line 22:		Line 22:
	if the sql file is compressed, extract it first.<br>		if the sql file is compressed, extract it first.<br>
	Now database and document root are finished, we need to work on apache next. The Webservers conf file can be setup with just 4 lines basically, being:<br>		Now database and document root are finished, we need to work on apache next. The Webservers conf file can be setup with just 4 lines basically, being:<br>
	<VirtualHost *:80>		<VirtualHost 127.0.0.1:80>
	DocumentRoot [path_to_the_htdocs_in_the_Document_Root]		DocumentRoot [path_to_the_htdocs_in_the_Document_Root]
	Alias /wiki [path_to_the_htdocs_in_the_Document_Root]/index.php		Alias /wiki [path_to_the_htdocs_in_the_Document_Root]/index.php

Maltfield: /* Setup OSEWiki from a Backup */

2018-05-27T18:07:57Z

Setup OSEWiki from a Backup

← Older revision		Revision as of 18:07, 27 May 2018
Line 27:		Line 27:
	</VirtualHost>		</VirtualHost>
	'''Don't forget the Alias'''<br>		'''Don't forget the Alias'''<br>
	'''Double-check that this apache vhost is not accessible to the public internet'''		'''Double-check that this apache vhost is not accessible to the public internet'''<br/>

	After that reload apache, and you should be able to reach the OSE wiki now through the ip of your server!<br>		After that reload apache, and you should be able to reach the OSE wiki now through the ip of your server!<br>