Installation of pygraphviz in MacOS 10.12 (Sierra)

Jarek Gadgets & hobbies No Comments

First attempt to the installation failed because of missing graphviz:

(django_python_env) MBP:mysite jhartman$ pip3 install pygraphviz
Collecting pygraphviz
 Downloading pygraphviz-1.3.1.zip (123kB)
 100% |████████████████████████████████| 133kB 1.4MB/s
Building wheels for collected packages: pygraphviz
 Running setup.py bdist_wheel for pygraphviz ... error
 Complete output from command /Users/jhartman/scripts/django_python_env/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/var/folders/t7/v05f2p5j5kgdd8t_px1lv1jc0000gn/T/pip-build-zn5secqs/pygraphviz/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /var/folders/t7/v05f2p5j5kgdd8t_px1lv1jc0000gn/T/tmp34txy01_pip-wheel- --python-tag cp36:
 running bdist_wheel
 running build
 running build_py
 creating build
 creating build/lib.macosx-10.12-x86_64-3.6
 creating build/lib.macosx-10.12-x86_64-3.6/pygraphviz
 copying pygraphviz/__init__.py -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz
 copying pygraphviz/agraph.py -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz
 copying pygraphviz/graphviz.py -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz
 copying pygraphviz/release.py -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz
 copying pygraphviz/version.py -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz
 creating build/lib.macosx-10.12-x86_64-3.6/pygraphviz/tests
 copying pygraphviz/tests/__init__.py -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz/tests
 copying pygraphviz/tests/test.py -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz/tests
 copying pygraphviz/tests/test_attribute_defaults.py -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz/tests
 copying pygraphviz/tests/test_attributes.py -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz/tests
 copying pygraphviz/tests/test_clear.py -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz/tests
 copying pygraphviz/tests/test_drawing.py -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz/tests
 copying pygraphviz/tests/test_edge_attributes.py -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz/tests
 copying pygraphviz/tests/test_graph.py -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz/tests
 copying pygraphviz/tests/test_html.py -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz/tests
 copying pygraphviz/tests/test_layout.py -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz/tests
 copying pygraphviz/tests/test_node_attributes.py -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz/tests
 copying pygraphviz/tests/test_readwrite.py -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz/tests
 copying pygraphviz/tests/test_string.py -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz/tests
 copying pygraphviz/tests/test_subgraph.py -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz/tests
 copying pygraphviz/tests/test_unicode.py -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz/tests
 running egg_info
 writing pygraphviz.egg-info/PKG-INFO
 writing dependency_links to pygraphviz.egg-info/dependency_links.txt
 writing top-level names to pygraphviz.egg-info/top_level.txt
 reading manifest file 'pygraphviz.egg-info/SOURCES.txt'
 reading manifest template 'MANIFEST.in'
 warning: no previously-included files matching '*~' found anywhere in distribution
 warning: no previously-included files matching '*.pyc' found anywhere in distribution
 warning: no previously-included files matching '.svn' found anywhere in distribution
 no previously-included directories found matching 'doc/build'
 writing manifest file 'pygraphviz.egg-info/SOURCES.txt'
 copying pygraphviz/graphviz.i -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz
 copying pygraphviz/graphviz_wrap.c -> build/lib.macosx-10.12-x86_64-3.6/pygraphviz
 running build_ext
 building 'pygraphviz._graphviz' extension
 creating build/temp.macosx-10.12-x86_64-3.6
 creating build/temp.macosx-10.12-x86_64-3.6/pygraphviz
 clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/include/python3.6m -c pygraphviz/graphviz_wrap.c -o build/temp.macosx-10.12-x86_64-3.6/pygraphviz/graphviz_wrap.o
 pygraphviz/graphviz_wrap.c:2954:10: fatal error: 'graphviz/cgraph.h' file not found
 #include "graphviz/cgraph.h"
 ^
 1 error generated.
 error: command 'clang' failed with exit status 1

Ok, so next obvious step was to install missing library using brew:

MBP:~ jhartman$ brew install libcgraph
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> Updated Formulae
ammonite-repl apache-flink checkstyle libass pcap_dnsproxy supervisor zanata-client
angular-cli bastet confuse neo4j s6 swiftformat

Error: No available formula with the name "libcgraph"
==> Searching for a previously deleted formula...
Error: No previously deleted formula found.
==> Searching for similarly named formulae...
Error: No similarly named formulae found.
==> Searching taps...
Error: No formulae found in taps.
MBP:~ jhartman$ brew install graphviz
==> Installing dependencies for graphviz: libtool, libpng, freetype, fontconfig, jpeg, libtiff, webp, gd
==> Installing graphviz dependency: libtool
==> Downloading https://homebrew.bintray.com/bottles/libtool-2.4.6_1.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring libtool-2.4.6_1.sierra.bottle.tar.gz
==> Using the sandbox
==> Caveats
In order to prevent conflicts with Apple's own libtool we have prepended a "g"
so, you have instead: glibtool and glibtoolize.
==> Summary
🍺 /usr/local/Cellar/libtool/2.4.6_1: 70 files, 3.7MB
==> Installing graphviz dependency: libpng
==> Downloading https://homebrew.bintray.com/bottles/libpng-1.6.29.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring libpng-1.6.29.sierra.bottle.tar.gz
🍺 /usr/local/Cellar/libpng/1.6.29: 26 files, 1.2MB
==> Installing graphviz dependency: freetype
==> Downloading https://homebrew.bintray.com/bottles/freetype-2.8.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring freetype-2.8.sierra.bottle.tar.gz
🍺 /usr/local/Cellar/freetype/2.8: 63 files, 2.6MB
==> Installing graphviz dependency: fontconfig
==> Downloading https://homebrew.bintray.com/bottles/fontconfig-2.12.1_2.sierra.bottle.1.tar.gz
######################################################################## 100.0%
==> Pouring fontconfig-2.12.1_2.sierra.bottle.1.tar.gz
==> Regenerating font cache, this may take a while
==> /usr/local/Cellar/fontconfig/2.12.1_2/bin/fc-cache -frv
🍺 /usr/local/Cellar/fontconfig/2.12.1_2: 487 files, 3.1MB
==> Installing graphviz dependency: jpeg
==> Downloading https://homebrew.bintray.com/bottles/jpeg-8d.sierra.bottle.2.tar.gz
######################################################################## 100.0%
==> Pouring jpeg-8d.sierra.bottle.2.tar.gz
🍺 /usr/local/Cellar/jpeg/8d: 19 files, 708.3KB
==> Installing graphviz dependency: libtiff
==> Downloading https://homebrew.bintray.com/bottles/libtiff-4.0.8.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring libtiff-4.0.8.sierra.bottle.tar.gz
🍺 /usr/local/Cellar/libtiff/4.0.8: 245 files, 3.4MB
==> Installing graphviz dependency: webp
==> Downloading https://homebrew.bintray.com/bottles/webp-0.6.0.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring webp-0.6.0.sierra.bottle.tar.gz
🍺 /usr/local/Cellar/webp/0.6.0: 36 files, 2.0MB
==> Installing graphviz dependency: gd
==> Downloading https://homebrew.bintray.com/bottles/gd-2.2.4_1.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring gd-2.2.4_1.sierra.bottle.tar.gz
🍺 /usr/local/Cellar/gd/2.2.4_1: 34 files, 1MB
==> Installing graphviz
==> Downloading https://homebrew.bintray.com/bottles/graphviz-2.40.1.sierra.bottle.1.tar.gz
######################################################################## 100.0%
==> Pouring graphviz-2.40.1.sierra.bottle.1.tar.gz
🍺 /usr/local/Cellar/graphviz/2.40.1: 536 files, 12.9MB

Everything looked find, however pip’ing still shown same problem with not installed library. I think the reason was in the warning show above:

In order to prevent conflicts with Apple's own libtool we have prepended a "g"
so, you have instead: glibtool and glibtoolize.

Ok, so where we have eventually the lib?

MBP:~ jhartman$ find /usr/local/Cellar/ -name "*graphviz*"
/usr/local/Cellar//graphviz
/usr/local/Cellar//graphviz/2.40.1/.brew/graphviz.rb
/usr/local/Cellar//graphviz/2.40.1/include/graphviz
/usr/local/Cellar//graphviz/2.40.1/include/graphviz/graphviz_version.h
/usr/local/Cellar//graphviz/2.40.1/lib/graphviz
/usr/local/Cellar//graphviz/2.40.1/share/graphviz
/usr/local/Cellar//graphviz/2.40.1/share/man/man7/graphviz.7

So we’re almost there. Let’s convince pip to use include and lib folder mentioned above (by enforcing it by --install-option):

(django_python_env) MBP:mysite jhartman$ pip3 install pygraphviz --install-option="--include-path=/usr/local/Cellar/graphviz/2.40.1/include/" --install-option="--library-path=/usr/local/Cellar/graphviz/2.40.1/lib/"
/Users/jhartman/scripts/django_python_env/lib/python3.6/site-packages/pip/commands/install.py:194: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
 cmdoptions.check_install_build_global(options)
Collecting pygraphviz
 Using cached pygraphviz-1.3.1.zip
Skipping bdist_wheel for pygraphviz, due to binaries being disabled for it.
Installing collected packages: pygraphviz
 Running setup.py install for pygraphviz ... done
Successfully installed pygraphviz-1.3.1

We’re done!

Read more

Fix for “Withings to Garmin Connect” authorisation problems

Jarek Gadgets & hobbies 2 Comments

It has been reported that the script does not work anymore:

jhartman@wieloryb:~/scripts/withings-garmin-v2$ ./sync.py
Traceback (most recent call last):
  File "./sync.py", line 124, in <module>
    main()
  File "./sync.py", line 59, in main
    sync(**opts.__dict__)
  File "./sync.py", line 116, in sync
    cookie = garmin.login(garmin_username, garmin_password)
  File "/home/jhartman/scripts/withings-garmin-v2/garmin.py", line 132, in login
    cookies = self._get_cookies(email=username, password=password)
  File "/home/jhartman/scripts/withings-garmin-v2/garmin.py", line 100, in _get_cookies
    data["lt"] = re.search("name=\"lt\"\s+value=\"([^\"]+)\"", preResp.text).groups(1)[0]
AttributeError: 'NoneType' object has no attribute 'groups'

I’ve managed to fix it. See latest commit in https://github.com/jaroslawhartman/withings-garmin-v2

Read more

Belkin KVM: Falied to validate certificate

Jarek Gadgets & hobbies, Telco and IT No Comments

After one of JVM updates, my Belkin KVM (Remote IP Manager) has started refusing to start with “Falied to validate certificate” error followed by “PKIX path validation failed”:

I’ve found following exceptions in Java console:

sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Algorithm constraints check failed on disabled signature algorithm: MD5withRSA
 at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:352)
 at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:249)
...

and

com.sun.deploy.security.BlockedException: User has denied the privileges to the code
 at sun.plugin2.applet.Plugin2ClassLoader.getPermissions(Unknown Source)
 at sun.plugin2.applet.Applet2ClassLoader.getPermissions(Unknown Source)
 at java.security.SecureClassLoader.getProtectionDomain(SecureClassLoader.java:206)
 at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
 at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
...

Google says that this is pretty common problem and returned plenty of receipts how to fix it.

Scanning the system for potential locations of the java.security file, there are four candidates. However, there is a catch: we should fix Java not in the system-wide location but the Java plugin for the web browser you’re using (Safari on MacOS in my case).

Jareks-MacBook-Pro:~ jhartman$ locate java.security
/Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/lib/security/java.security
/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/lib/security/java.security
/Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home/jre/lib/security/java.security
/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/security/java.security

So we can skip the last three locations and just focus on the 1st one. Change the lines as described

Jareks-MacBook-Pro:~ jhartman$ edit "/Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/lib/security/java.security"

jdk.certpath.disabledAlgorithms=MD2, RSA keySize < 256, \
 DSA keySize < 1024, EC keySize < 224
jdk.tls.disabledAlgorithms=SSLv3, RC4, DH keySize < 768, \
 EC keySize < 224

Save and restart the browser (Safari). This time the KVM applet can start (after plenty of security warnings though).

Update

The application has to be also white-listed, otherwise it will generate error as below:

Open System Preferences, Java settings and add IP address of your KVM to the list:

Add following:

Again: save and restart your browser.

Update 2

Another security exception is required (after one of Java 8 updates):

MBP:~ jhartman$ edit "/Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/lib/security/java.policy"

Add this line within the grant { }; section:

// KVM
permission java.net.SocketPermission "192.168.1.111:444", "connect, accept"; 

Read more

How to check if all Calibre books are already in the Amazon cloud?

Jarek Gadgets & hobbies No Comments

Preface

My primary software for managing Kindle books is Calibre. I’m using Amazon cloud for books synchronisation and sometimes I’d like to ensure that all my books stored in Calibre have been already sent to the cloud. Mainly, to avoid re-sending books and avoid duplicates.

Getting a list of Amazon books

Refer to Kindle Library List article.

After executing this step you should have one or more text listing your books:

Jareks-MacBook-Pro:Downloads jhartman$ java -jar KindleLibrary.jar 1.htm
Amazon book list extractor
Elements found:400
Saving 1.html
Saving 1.txt
Saving 1.xml
Done!
Jareks-MacBook-Pro:Downloads jhartman$ java -jar KindleLibrary.jar 2.htm
Amazon book list extractor
Elements found:400
Saving 2.html
Saving 2.txt
Saving 2.xml
Done!
Jareks-MacBook-Pro:Downloads jhartman$ java -jar KindleLibrary.jar 3.htm
Amazon book list extractor
Elements found:373
Saving 3.html
Saving 3.txt
Saving 3.xml
Done!

Getting a list of Calibre books

Now open your Calibre, press a small triangle icon next to the Convert Books icon. Follow the instructions below:

After pressing OK you should get your Calibre books:

Jareks-MacBook-Pro:Downloads jhartman$ head CalibreBooks.csv
title,authors
"Świat pani Malinowskiej ; Trzecia płeć","Tadeusz Dołęga-Mostowicz"
"Drugie życie doktora Murka","Tadeusz Dołęga-Mostowicz"
"Świat pani Malinowskiej","Tadeusz Dołęga-Mostowicz"
"Wysokie Progi","Tadeusz Dołęga-Mostowicz"
"Złota maska","Tadeusz Dołęga-Mostowicz"
"Morderstwo pod cenzura","Marcin Wroński"
"Wąż Marlo t1","Marcin Wroński"

Compare the lists

In the very last step, use tide up the output files.

#!/usr/bin/env bash

awk -F $'\t' '{print $1,$2}' 1.txt > Amazon.tmp
awk -F $'\t' '{print $1,$2}' 2.txt >> Amazon.tmp
awk -F $'\t' '{print $1,$2}' 3.txt >> Amazon.tmp
sort Amazon.tmp > Amazon.txt

awk -F ',' '{print $1,$2}' CalibreBooks.csv | sed 's/"//g' | sort | grep -v "title authors" > Calibre.txt

And finally – use a side-to-side text compare tool (JEdit, WinMerge, any other) and view the differences between Amazon.txt and Calibre.txt:

Review the differences and take proper actions:

  • Send the file to the cloud
  • Remove duplicates in the cloud (yes, this will be also visible in the output)

I hope it will help.

(more…)

Read more

Alfred 3 workflow – Time Zone

Jarek Gadgets & hobbies 2 Comments

Do you like Alfred? I like too… I’ve recently found bunch of very nice Workflows. For me, the most used one is for sure Time Zone.

The workflow has been created by Mr Carlos from New Zealand and published on the Alfred Forum. I’ve enhanced the workflow by adding a few new functionalities:

 

 

 

  • Migration to Alfred 3
  • Converting any time to all pre-defined time zones
  • Enabling 24-hrs time format
  • Displaying phone country code

After all it looks like below:

Conversion of a time in your local time to your pre-defined time-zones – “tz <time>” in formats HH, HHMM or HH:MM:

CarlosNZ, thanks for this great tool!

Workflow can be downloaded from here: Alfred-TimeZones

(more…)

Read more

NoSQL: Exiting inner Replica loop with exception com.sleepycat.je.rep.RollbackProhibitedException

Jarek Telco and IT No Comments

If a NoSQL Storage Node is not running for a while, after startup it may complaining:

2016-11-17 13:22:34.934 UTC WARNING [admin1] JE: Exiting inner Replica loop with exception com.sleepycat.je.rep.RollbackProhibitedException: (JE 6.4.15) 1(1):/opt/ece/data/nosql/storage2/kvroot/ECEStore/sn1/admin1/env Node 1(1):/opt/ece/data/nosql/storage2/kvroot/ECEStore/sn1/admin1/env must rollback 41 commits to the earliest point indicated by transaction id=-392 time=2016-11-17 14:18:53.969 vlsn=884 lsn=0x0/0x6e4be1 in order to rejoin the replication group, but the transaction rollback limit of 10 prohibits this. Either increase the property je.rep.txnRollbackLimit to a value larger than 10 to permit automatic rollback, or manually remove the problematic transactions. To do a manual removal, truncate the log to file 00000000.jdb, offset 0x6e4944, vlsn 881 using the directions in com.sleepycat.je.util.DbTruncateLog.  ROLLBACK_PROHIBITED: Node would like to roll back past committed transactions, but would exceed the limit specified by je.rep.txnRollbackLimit. Manual intervention required. Environment is invalid and must be closed.

(more…)

Read more