First BangPypers Meetup

BangPypers , the Bangalore Python User Group is one of the largest and oldest Python User Group in India. It has been running successfully for more than 10+ years. Most of the known faces in Indian Python Community have been a part of this meetup group at sometime or other.

After I moved to Bangalore I have been on the look out to attend the meetups but I missed it in the first 2 months since some work used to come by on the meetup days. Last Saturday, 21st September, I finally made time to attend my first BangPypers meetup.

The meetup was schedule at 10.30 am at Visa Technology Centre, Bagmane Tech Park. I decided to go by Cycling since I was also looking for a long ride in my Cycle for a while. But alas it took longer than expected owing to a wrong turn in my way.

The theme of this week was Design Thinking and it had 3 Talks scheduled. I reached around 11.15 am almost missing the first Talk which was a Design Patterns 101 Talk. The Speaker was almost in his closing notes. Nevertheless I found a place to seat in the almost full room.

I took out my phone to take notes and waited for the next talk which was on Design Pattern for distributed Architecture . The Talk covered various points on best practices related to distributed systems and tools and techniques to achieve them. Discussion revolved around Horizontal Scalability, Responsiveness, Security, Failure Handling, Centralised Logging, Metrics, Request Tracing, Health Checks, Configuration and Discovery. Since I never dealt with production level Python applications or Distributed systems the terms were quite new to me, few of them I encountered in past but didn’t quite fiddle in depth. I made a note of the discussion for future reference.

The last talk of the day was Organize your bookshelf using Micropython by Vinay Keerthi who was one of the Hosts at Visa, which is also a PyCon India 2019 accepted Talk. He made a LED book shelf organiser which would tell him the position of a book in the shelf by lighting LED array. He used Micropython on a NodeMcu in which he built a simple Flask app which would fetch data from a Postgresql database and store it in a queue which would send signals to the LED. I really liked the idea and it was a really hacky way to find a book. After his presentation I talked to him to get some pointers on starting my first Micropython project which I plan to do in the upcoming days.

After the session I talked to the organisers Anirudha and Abhiram with whom I shared my experiences at Hyderabad Python Community and also how they conduct BangPypers meetups. Visa had arranged lunch for the attendees so the discussion continued at the Lunch table where I got to talk with other attendees and interacted about the work they were doing. Bangalore has a vibrant developer community and there are many more communities like PyData, PyLadies which conduct regular meetups. PyLadies Bangalore are going to have a meetup on coming Saturday, September 28. Here is the announcment about the same.

At the end Vinay shared some more stories of his Hacks and experiences with tinkering with Micropython. And yeah here are some photos of the meetup I took

The thing that made the meetup nice was the great Venue and the Topics of the Talks. With the boom of ML/AI in Python Land having Talks that are unlike those, but most fundamental topics is good. Hope to attend future meetups as well!

Lastly I rode back home completing my first 20km streak in Cycle 😛

I also tweeted about the meetup after I returned home that day.

Takeways from the Meetup

  • Design Patterns in Python – Spend more time learning these.
  • Got to know about best practices in distributed system – Keep notes for futute projects.
  • Micropython getting started – Order some NodeMcu and get the hands dirty. TO DO: Read blogs regarding starter projects.

DevConf India 2019 experience

One month ago I attended DevConf India 2019. It was held from August 2-3 at Christ University, Bangalore. It was quite a while since I attended a conference, the last one being PyCon India 2018 . Due to my laziness in writing I have altogether missed writing my Conference experiences till now. This will be in fact my first Conference blog post. Now on I will make sure to write about each of the conferences I go.

I came to know about DevConf India last year when it was first held, from people in my developer circle. DevConf India is organised by Red Hat and has similar events in US and Czech Republic. Since I was in Bangalore this year I made sure I attend it after I saw the dates in Twitter. I registered as soon as I came to know about it.

Day 1

I mostly planned on meeting up with people and attend a few talks. I started at 8.30 am from my place and unfortunately missed the Keynote owing to a bad experience with a Bike Taxi service. I reached the Venue around 10 am and collected my Attendee badge and T-shirt. Then I headed towards the Keynote Session Hall where I met Naren from Chennaipy. I earlier met him at PyCon India 2018. It was nice catching up with him.

After having breakfast at the venue I headed to the Booth area where I met Chandan . I started visiting the booths asking questions about various projects like Fedora, Debian, CentOS. Shortly after I met up with some more familiar faces from DgplugSayan, Rayan , all of them I met during PyCon India last year. I expected a Dgplug staircase meeting at DevConf but unlike last year there were less attendees this time. After that we went for lunch at the Cafeteria where I met Ramki , Praveen and pjp . Few days earlier I was reading pjp’s tutorial regarding gcc and gdb from dpglug irc logs . It was nice to catch up with him in person. I was discussing with Sayan and Praveen about the initial days of dgplug at my college at NIT Durgapur, attending their first talk in 2013 when I had just joined my college.

After Lunch I decided to attend few talks. I attended a talk regarding Evolution of Containers – there I came across terms like Chroot, Cgroups, Namespaces , how the whole container ecosystem was born. I have been always been inquisitive about containers and though I haven’t really worked on containers before this talk really fascinated me to dive into the world of containers.

Then I attended a talk on What hurts people in Open Source Community . The talk helped to set my expectations right regarding contribution to a Open source project and Community.

After that I went to the Closing Keynote of the day shortly after which we went for evening snacks where we had more discussions over Coffee and Dosa – we noticed a item mentioning ‘Open Dosa’ over which we laughed a lot 😛 . And it was finally close of the day.

Day 2

I reached a little late to the venue and went straight to the talk that I didn’t want to miss. It was a Documentation BoF where speakers were discussing how to create effective documentation and tools for creating collaborative documentation. I came across User Story based documentation and tools of the trade like asciidoc and asciidoctor . I met Ladar Levison there during that session and talked with him regarding better project organisation. He gave me his business Card which mentioned Lavabit . Little did I know about him until I read this article which explained more about Lavabit and his role in Snowden’s secure email communication. But that was after this conference and I wished I could talk more about Privacy and Lavabit projects.

After that I went for lunch with Sayan, Chandan and Rayan where we chatted on lot of different stuff on open source, food and conferences. After Lunch I went to attend Sinny‘s talk on Fedora CoreOS whom Sayan introduced last day.

Finally it was nearing the end of the day. I went to attend the closing keynote by Jered Floyd and sat beside Christian Heimes from Red Hat who was sharing anecdotes from his travel experiences.

Notes from the Conference

I made few notes that I would like to share from my experience at the Conference and also as a note to me for future Conferences

  • Try to search about people you meet so that you can know more about them. You may not know everything about the person you are talking with. But actually the person can be a mine of knowledge. Ask for the person’s email/Twitter so that you can follow up on email or Twitter after the conference.
  • It’s always good to prepare some questions if you are likely to meet a person you met/knew online. You have the opportunity to talk face to face and ask about the projects the person works on. You can even do that within the conference when you are free.
  • When you attend a Talk ask good questions that can start a conversation. Usually people take a interest in following up after the Talk with you and you get to talk to more people.
  • It’s always good to be Speaker at the Conference. That way there is higher chances that you can start a conversation with people you don’t know and meeting for the first time. This is something that I really need to work on and hopefully I will be able to submit a talk in the next Conference I attend.
  • When going for Lunch tag along with a group so that you get meet more people. If you are an introvert this really works well as you can meet friends of friends and you can interact much easily!

And yes don’t forget to take pictures 🙂 It really bring memories. It may sound weird but this is something I really forget every time I meet up with people and later wait for Conference photos.

A New City, A New Beginning

Two and a half months earlier I shifted to Bangalore after a 2 years stint at my first Company at Hyderabad. I was looking for new opportunities and started appearing in interviews when I hit the hard realisation that what I learned till now was not enough. I need to self-learn more and interact more with people and learn from stories that people experienced. I landed up a job at a EDA based firm at Bangalore and decided to move in there.

Leaving the city, but not the memories !

Time really flies and it had been already 2 years working in the Software Industry. I met with a lot of people, made friends and shared some good experiences. I became aware of the meetup culture, realised writing code is not the only way that makes you a good developer, realised that communication is a key factor in conveying your ideas.

I became part of the HydPy Community and organised 2 conferences – PyConf Hyderabad 2017 and PyCon India 2018 . I met some like minded people passionate about technology and flourishing the Python Community at Hyderabad. I still am a part of the Community and hope to continue doing so.

I also attended the dgplug Summer Training in 2018, made many friends there as well whom I keep meeting during conferences. They are a really amazing community and there is always something to learn from each of the IRC conversations.

I interacted with a lot of people from Python and open source Community. Going to Conferences and meetup is really a good way to interact with some awesome people and sharing their knowledge. But it’s also true that you really need to work on something tangible and develop your skills. Then only you can move forward and also put the experiences to use.

I have been living in the heart of the fast growing city at Hyderabad. It’s been a nice experience here as a whole (except from traffic on rainy days!). I explored to places around in the initial days. Some significant events happened during my stay – the GES 2018 was held, Hyderabad metro rail was inaugurated, the first IKEA store in India opened up . I Spent my first Durga puja away from home. Realised that Kolkata Biryani is still better than Hyderabadi Biryani. Saw more buildings, Tech parks and flyovers being constructed in Rapid phase. Got to experience the hot summer and pleasant winters. All together it had been a good time in the city of Nizams. Hope to come again here someday !

The New Phase, What Next ?

It’s been 2.5 months while I came here at Bangalore. It’s called the Garden City of India because of the amount of greenery it has. But I would rather call it a City of Traffic ! Dealing with traffic can be terrifying here. Nevertheless there is a lot of greenery left and I see palm trees here and there quite often. The area I live is booming with an array of multinational companies, I also observed more number of semiconductor based companies. Rightly calling it “Silicon Valley of India”. It’s pretty early to state my experience in this city so maybe I will write about it in the next phase 🙂

So what next ? I want to make my time here more productive and also develop some good habits that I have been lingering about. Get into the habit of writing, take time out for more self-learning, contribute to open source more often, interact with more people, put the experiences to use as much as I can, get more exercise and cut laziness. In fact I got me a cycle and have been doing my daily commute with it! I know now it’s more talk than work now. But I want this blog post as a reminder so that whenever I wander off I can come to this page and find what I need to do! And keep myself prepared for the next phase.

Adding Print Preview Support to CEF

Chromium Embedded Framework (CEF) is a framework for embedding Chromium-based browsers in other applications. Chromium itself isn’t a extensible library. It helps you to embed a chromium browser in a native desktop application. The project was started by Marshall Greenblatt in 2009 as an open source project and since then it has been used by a number of popular applications like Spotify, Amazon Music, Unreal Engine, Adobe Acrobat, Steam and many more (full list can be found in this wiki). CEF is supported for Windows, Linux and macOS platforms.

There are 2 versions of CEF – CEF1 and CEF3. CEF1 is a single process implementation based in Chrome Webkit API. It’s no longer supported or maintained. CEF3 is a multiprocess implementation based on Chromium Content API and has performance similar to Google Chrome.

Preface

The purpose of writing this article is to document the work that I did while working on the CEF project. The major focus is on the Print Preview addition to CEF upstream project where I worked on.

For the past 1 year I have been working on CEF as a part of my day job. Initially the work was to maintain updates of CEF with every Chromium version released. We had a fork of the CEF open source project in which we applied some extra patches as per the requirements of the custom desktop application that used it. Building CEF was quite similar to building Chromium. It had a build script which was mostly used from downloading the code, building and packaging the binaries. Everything is documented here in this link . Upgrading the CEF was quite some task since building CEF took a lot of time and resources (a lot of CPU cores and a lot of Memory) and since CEF was based out of chromium I had to skim through parts of chromium code. Chromium has a nice developer documentation and a good code search engine that eased a lot of things. But owing to the fact that chromium has a huge codebase the documentation was outdated in few areas.

Feature Description

The interesting part of the CEF project came when I was handed over a work of a missing piece in CEF. Chromium supports in browser print preview where you can preview pages before printing, similar to one shown in the picture below.

CEF didn’t support this feature and had the legacy print menu where you cannot preview the pages to be printed.

This meant applications that used CEF couldn’t support print preview within them. The task was to make print preview available in CEF.

Initial work

The work started with CEF version 3112 (supported chromium v60) and was in a working state in CEF 3239 (supported chromium v63) in our CEF fork. Then the change was supported only in Windows since our desktop application that used it was a Windows only application. I was handed over the work in CEF 3325 (supported chromium v65) where the following specs already existed in the Print Preview patch. The relevant blocks of code is available in the CEF source code now.

  • enable_service_discovery is disabled now
  • CefPrintViewManager::PrintPreviewNow() will handle print-preview
  • CefPrintViewManager will not handle the print now. It handles the PrintToPdf function exposed through browserHost. Also, it listens to two print preview message PrintHostMsg_RequestPrintPreview, PrintHostMsg_ShowScriptedPrintPreview and generate the browserInfo for print preview dialog
  • Define interfaces from web_modal::WebContentsModalDialogManagerDelegate and web_modal::WebContentsModalDialogHost required by the constrained window of print preview dialog
  • Define platform specific definitions of GetDialogPosition() and GetMaximumDialogSize() used by browser platform_delegate
  • Register Profile preferences required for printing
  • Add ConstrainerWindowsViewsClient class which is called in browser_main.cc
  • CefPrintViewManager::InitializePrintPreview() initializes the print preview dialog which further calls PrintPreviewHelper::Initialize() which generates the browser_info required by print preview
  • Remove CefPrintViewManagerBase and its associated methods from CefPrintViewManager. Those methods are redundant after the print preview changes
  • Check for switches::kDisablePrintPreview in CefPrintRenderFrameHelperDelegate::IsPrintPreviewEnabled() to determine whether print preview is enabled
  • Add chromium patch to fix errors in debug build

My Contribution to CEF Print Preview

After I took over the CEF print preview work I did a number of changes to print preview, specs of which is documented below

  • Add print_preview_resources.pak in BUILD.gn to fix blank screen error which came from chromium v72 onwards because of updated print preview UI
  • Add PrintPreviewEnabled() to extensions_util
  • Add switches::kDisablePrintPreview to kSwitchNames in CefContentBrowserClient::AppendExtraCommandLineSwitches() to disable print preview on using --disable-print-preview command line switch
  • Remove print_header_footer_1478_1565 chromium patch since it’s no longer required to disable print preview by default
  • Add WebContentsDialogHelper() to wrap web_modal::WebContentsModalDialogManagerDelegate and web_modal::WebContentsModalDialogHost interface methods. Move it in a separate header and cc file
  • Add support for disabling print preview via chromium preferences
  • Disable print preview for OSR mode
    • For ui print this is done by using CefBrowserHostImpl::IsWindowless() method
    • For scripted print this is done by passing the is_windowless parameter in CefPrintRenderFrameHelperDelegate object from CefContentRendererClient::MaybeCreateBrowser() method in content_renderer_client.cc
  • Fix DownloadPrefs::FromBrowserContext for CEF use case. Add GetDownloadPrefs() to CefBrowserContext and use that to get the download_prefs in chromium
  • Add extra_info param to CreatePopupBrowserInfo()
  • Fix MacOS build. Add GetNativeView() method for MacOS platform delegate and update GetMaximumDialogSize() method
  • Disable print preview for MacOS (in extensions::PrintPreviewEnabled()) since the print dialog crashes on print
  • Disable print preview if pdf extension is not enabled
  • Use CEF file chooser instead of chromium while saving to pdf in print preview. Add ShowCefSaveAsDialog() and SaveAsDialogDismissed() to PdfPrinterHandler.

Challenges faced

Integrating print preview was a big and non-trivial change in CEF since not only it needed good understanding of the printing code in chromium but also the print preview feature was getting constant updates from Chromium. The code was constantly changing with every chromium version released and the print preview chromium documentation was outdated.

CEF3 has a multiprocess architecture similar to chromium’s documented here . There is a main browser process and multiple renderer processes. Debugging multiprocess applications can be trickier. I used Visual Studio which made things a bit easier as it has the Child process Debugging power tool which is a extension that would automatically attach Child processes and helped to debug into the child processes whenever they spawned up.

The chromium v72 version introduced a new print preview UI which broke the renderer, we got a blank screen in print previw. It took weeks to figure out what was wrong. Finally it came out that a pak file was missing which needed to be included in BUILD.gn. I had to spend multiple debugging session with my team to figure that out.

Also it had to be supported for all platforms (Windows, Linux, macOS) to qualify to be merged to the CEF upstream repo. Each platform had a different way of rendering dialogs. Though the windows support was working the Linux and MacOS weren’t supported in the changes yet. I added the support for Linux platform after building CEF in a linux VM. The MacOS support finally didn’t work out and we had to keep using legacy print for Mac platform. Though I needed to ensure the change built fine in Mac, so I had to build it for Mac as well (I was given a separate Mac machine just because of this since Mac doesn’t ship VM images) and in fact the change broke the MacOS build so the issues had to be fixed.

Conclusion

Even after all these changes the functionality broke after a architectural change was made in CEF in version 3770 (supported chromium v75) in this commit which rendered blank screen during print preview. Marshall took over the work from there and made a number of changes in the patch which can be seen in this PR chromiumembedded/cef#126 . The change was added manually in master revision 1669c0a on 20th July. It will be supported from the next CEF version (supported chromium v77). The current implementation supports print preview in Windows and Linux platforms via a --enable-print-preview flag.

Overall it has been a good experience working in the project and I got to know a lot about chromium itself. This was my first major contribution in a C++ based project. It helped me to understand how a browser works under the hood, how a browser renders web pages and processes javascript. I hope to carry forward this knowledge in some similar browser based project in future.

Rescuing GRUB2 from rescue mode in Fedora 30

About 3 months back I installed the newly released Fedora 30 OS – dual boot with Windows 10 in my PC. This blog post comes from the notes I made during that time and as I troubleshooting note for future.

I had Fedora 29 and Windows 10 dual boot in my PC before that. The Fedora install partition was running out of space due to low disk space selected during my last install so I decided to do a clean reinstall this time. I made a live usb using the Fedora media writer for windows and the Fedora 30 iso available at getfedora download page. I followed the usual steps that I followed for installing earlier Linux installations in my PC, similar to what mentioned in this video.

The installation went fine and finally I was ready to boot from my hard drive. Then I saw what is called the Dreaded GRUB2 boot prompt.

error: unknown filesystem.
Entering rescue mode...
grub rescue>

First Attempt for Fix

I quickly started finding way to fix the grub. The first thing I found was the steps listed in this video. I had to choose the right partition from which the bootloader would load.

grub rescue> ls
(hd0) (hd0,msdos4) (hd0,msdos3) (hd0,msdos2) (hd0,msdos1)

This shows the various partitions in my hard drive. One of this is the linux partition where my Fedora 30 OS is installed. I need to list all the partitions and one of them will have the linux filesystem.

grub rescue> ls (hd0, msdos4)/
bin/  boot/  dev/  etc/  home/  lib/  lib64/  lost+found/  media/  mnt/  opt/  proc/  root/  run/  sbin/  srv/  sys/  tmp/  usr/  var/

Now I ran the following commands and waited for the system to boot up

grub rescue> set prefix=(hd0, msdos4)/boot/grub2
grub rescue> insmod normal
grub rescue> normal

I got the same grub rescue boot prompt but this time with a different error

error: file '/boot/grub2/i386-pc/normal.mod' not found.
Entering rescue mode…
grub rescue>

Second Attempt ..

The issue was that, the i386-pc folder was missing in the /boot/grub2 folder. The fix that I found for the issue was related to grub2 not being properly installed at the boot location. Luckily I was able to boot Fedora from UEFI Boot option from the Boot Menu. I logged into Fedora and reinstalled grub2.

$ sudo grub2-install /dev/sda
$ dnf install grub2-efi

I hoped that this would fix the issue, but it again came down to the same starting point loading the grub2 rescue prompt.

Third time is the charm !

I further searched and landed up in the fedora grub2 manual . After reading it I realized there is something wrong in my grub2 configuration. I booted into my OS using UEFI boot and opened /boot/grub2/grub.cfg file. The entry for Windows was missing. I followed the steps given in this section. I went to the grub rescue prompt and fired the following commands

grub rescue> set root=(hd0, msdos4)
grub rescue> linux /boot/vmlinuz-5.1.20-300.fc30.x86_64 root=/dev/sda4
grub rescue> initrd /boot/initramfs-5.1.20-300.fc30.x86_64.img
grub rescue> boot

Then I recreated the grub.cfg file using these commands

$ grub2-mkconfig -o /boot/grub2/grub.cfg
$ grub2-install --boot-directory=/boot /dev/sda

Voila ! I was able to see the grub menu with all the boot entries.

Postmortem Report

So why did the issue actually occur ? It didn’t happend in the past whenever I did a fresh installation nor it’s a issue specific to Fedora 30. I tried to dig the actual cause of the issue and after a little finding and introspection I came to this conclusion.

Using the Fedora media writer was in fact the place where I unknowingly did a mistake. I usually used UNetbootin for creating my linux live usb in past which made the images to boot in BIOS only mode. The Fedora media writer enables to boot in both UEFI and BIOS mode. My Windows installation boots via Legacy boot and it has always been like that and since I have been using UNetbootin earlier, it always picked up Boot via BIOS for the live images. This time while creating the Fedora 30 image using Fedora Media writer the default boot mode picked up was UEFI and that created a EFI GRUB2 configuration. Now when I booted the live usb I just booted from the “Boot from USB” option without noticing whether it was UEFI or BIOS  and I went ahead and did Fedora 30 installation. Now my default boot option was Legacy Boot (since it supports the Windows boot) while the installed Fedora OS grub loader was created to boot in EFI mode. That in turn caused this problem due to a corrupted grub2 configuration.

Moral of this Story

Always be careful while created OS images. Check how it is supposed to boot. In case of dual boots all the OS must boot via the same mode – either both UEFI or both BIOS. So make sure when you are doing a clean install of the second OS it must boot via same mode as the already installed OS.

GSoC 2016 Wrap-up:The End of a wonderful Journey

So GSoC 2016 comes to an end as the thirteenth week in the Coding period wraps up! Well actually it is way beyond these 13 weeks when it started. It was actually in the beginning of March when I started interacting with PublicLab. It’s been a wonderful experience working so far. The place where my actual Open-source journey began. And I have learnt so much in the way.

Final Works

Here is my final report that I will submit for the Final evaluation:

GSoC 2016: Final Work Product of Expanded Q & A System for publiclab.org

This research note contains the detailed report of my work and  the contributions I made.

I also want to show my contribution graph here

contribution

By the way you can find me on Github with my username @ananyo2012

The design changes are merged and I also managed to make some contribution to the Rich Editor. Here is my PR #40 in the PublicLab.Editor repo though it is not merged yet. I really need to learn nodejs before I make any significant contributions to the Rich Editor. Also I made an Rich Editor update in plots2 in PR #664. I spent the week doing fixes on small and some large bugs a creeped up. Also I wrote a wiki on Q & A system and made the final research note for my evaluation.

Also as my mentor insisted I made some first-timer issues in plots2 that could be taken up by new contributors who are completely new to open source. This was a great move to welcome new contributors to our codebase and applause to PublicLab for doing such a great job!

And finally it was all set to go for the Final Evaluation! Results would come by August 30th!

Tough times and Lesson learnt

There were some breaking changes after the code got deployed. It was due to the PR #600 I made. I started working on this long ago and I hadn’t mentioned about it in my any of my blog posts since I wasn’t sure about how it would turn up. It was a work on updating the slugs for research notes and wikis using friendly_id. But things were tough right from the beginning since there was already a diverse slugging system present and I had to make changes keeping the format of the slugs intact. On deploying the code the old slugs of the notes and wikis got updated to new slugs and the older ones were no longer available. So all previous links failed that pointed to those research notes. The new slugs were supposed to redirect to new ones as done using friendly_id. It used a friendly_id_slug table to store the old urls but unfortunately due to some issues they weren’t saved as expected, also I missed few test cases which didn’t predict this case while testing and it was a complete disaster. But fortunately PublicLab had good database backups and things were reverted back to normal in no time.  Some code from the PR had to reverted back and the issue was fixed.

Moral of the story:  Always write good tests and think of rigorous test cases before deploying any code. In fact good tests are the lifeline of a good software development cycle.

Experiences and Best Moments

Well it was an really good experience overall! I really learnt a lot throughout the entire summer. The idea of working alongside with so many people even when you are distances apart is really amazing! We had a Video call with our mentors and other GSoC students at the end of last month and it was great! And people in PublicLab were very helpful giving us reviews in research notes alongside with our mentors.

Thanks to PublicLab and Google OSPO for giving me this wonderful opportunity! Hope to participate in GSoC 2017 again!