Pivotal Engineering Journal

Technical articles from Pivotal engineers.

    • Jean de Klerk
    Headless UI Testing with Go, Agouti, and Chrome Acceptance tests for your UI are an excellent way to cover user functionality. Let’s see how to write a simple acceptance test in Go with Agouti and have it run headlessly in a CI environment with Chrome. Categories:   Testing    Go    Agouti    Chrome    UI Testing    UI   
    • Brian Cunnie
    • Lyle Franklin
    • Chris Dutra
    Leveraging NSX's Features with BOSH's vSphere CPI BOSH, a VM (Virtual Machine) orchestrator, includes the ability to interoperate with NSX, a network virtualization platform, when deploying to a vSphere IaaS (Infrastructure as a Service). This blog post describes deploying VMs as the backend of an NSX Load Balancer. Categories:   BOSH    NSX    vSphere   
    • Venkatesh Raghavan
    • Jesse Zhang
    • Haisheng Yuan
    • Omer Arap
    • Xin Zhang
    • Ivan Novick
    Profiling Query Compilation Time with GPORCA GPORCA is Pivotal’s Query Optimizer for Greenplum Database and Apache HAWQ (incubating). In this post, we describe how users can profile query compilation with GPORCA. This will aid users in understanding which of GPORCA’s steps is the most resource intensive, and what transformations are being triggered. Based on this information, users can provide query hints to reduce or increase the search space, see where the time and memory is being spent, and learn how to influence its decision making. Categories:   Databases    GPDB    HAWQ    PQO    GPORCA    Query Optimization    SQL   
    • Heather Farley
    • Ian Fisher
    Everything and the Spring Cloud Data Flow Sink Get started with Spring Cloud Data Flow Streams by creating a custom Sink app and deploying to Pivotal Cloud Foundry. Categories:   Spring Cloud Data Flow    Spring Cloud Data Flow Streams   
    • Brian Cunnie
    • Lyle Franklin
    How to Customize a BOSH Stemcell BOSH Stemcells are Linux-based bootable disk images upon which BOSH applications may be deployed. This blog post describes a process to customize a stemcell (most often used to troubleshoot stemcell boot problems). Categories:   BOSH   
    • Brian Cunnie
    Updating a BOSH Release Authors of a BOSH Release may want to release a new version when the upstream application is updated. This blog post describes the process of updating a BOSH Release while avoiding common pitfalls. Categories:   BOSH   
    • Dat Tran
    • Megha Agarwal
    Test-Driven Development for Data Science Unravelling Test-Driven Development for Data Science. Categories:   TDD    Data Science    Machine Learning    Agile    Pair Programming   
    • Benjamin Tan Wei Hao
    How to Set up a Distributed Elixir Cluster on Amazon EC2 Learn how to set up an Elixir cluster and how to deploy a Phoenix application on Amazon EC2. The techniques outlined in this article can equally apply to other providers such as Digital Ocean and Linode. Categories:   Elixir    Distributed    Cluster    Phoenix    Edeliver    Deployment    Erlang   
    • Nikos Armenatzoglou
    • Karthikeyan Jambu Rajaraman
    • Shreedhar A Hardikar
    • Craig Chasseur
    • Foyzur Rahman
    • George Caragea
    • Venkatesh Raghavan
    • Addison Huddy
    • Xin Zhang
    Improving Query Execution Speed via Code Generation A code generation based solution inside the GPDB execution engine. Categories:   GREENPLUM DATABASE    QUERY EXECUTION    CODE GENERATION   
    • Brian Cunnie
    Concourse has Badges Use Concourse’s badges to display the health of your project Categories:   Concourse   
    • Brian Cunnie
    • Forrest Sill
    Concourse without a Load Balancer nginx is a less-expensive alternative to a load balancer for a BOSH-deployed Concourse server’s SSL termination. Categories:   BOSH    Concourse   
    • Daniel Gustafsson
    GPDB merge with PostgreSQL 8.3 Greenplum merge with PostgreSQL 8.3 Categories:   PostgreSQL    Greenplum Database    Databases   
    • Srivatsan Ramanujam
    • Regunathan Radhakrishnan
    • Jin Yu
    • Kaushik Das
    Operationalizing Data Science Models on the Pivotal Stack Categories:   Data Science    Greenplum    SCDF    PCF    GemFire   
    • Ben Grohbiel
    Writing an Ionic 2 Application for Production On a recent Pivotal Labs engagement, we built a hybrid application using Ionic 2. This post provides some technical considerations for building your own. Categories:   Mobile    Hybrid Apps    Ionic   
    • Morgan Fine
    • Joseph Palermo
    • Chris Hendrix
    • Natalie J. Bennett
    Faster Pipelines With Compiled BOSH Releases Compile once, deploy many times. Categories:   BOSH    Concourse    Testing   
    • Marc Spehlmann
    Improving Constraints In ORCA ORCA is Pivotal’s Query Optimizer for big data. We look at how we improved ORCA’s understanding of logical constraints. Categories:   Databases    Query Optimization    SQL   
    • Sam Smith
    • Gabriel Ramirez
    • James Wen
    • Jacques Chester
    Creating a Custom Buildpack This article will describe how to create a custom buildpack using Rust as an example language. Categories:   Cloud Foundry    Buildpacks    Rust    CF Runtime   
    • Dat Tran
    • Alicia Bozyk
    API First for Data Science How API first can help to create smart data-driven apps. Categories:   Data Science    Machine Learning    API First    Cloud Foundry    Smart Apps   
    • Gabe Hollombe
    Building a Native Navigation Menu for iOS with Turbolinks 5 Using Turbolinks 5 to hide your site’s HTML navigation and present a native navigation view to mobile app users. Categories:   Rails    Turbolinks    iOS   
    • Dylan Griffith
    Using Action Cable With Cloud Foundry A guide to configuring and deploying a Rails 5 Action Cable app to Cloud Foundry. Categories:   Ruby On Rails    Cloud Foundry   
    • Sai To Yeung
    Why you should stub, not shallow render, child components when testing React A better way to avoid brittle unit tests in React Categories:   TDD    React    Javascript   
    • Konstantin Semenov
    Data Driven Testing with Spek Finding a usable approach to data-driven testing with Spek Categories:   Kotlin    TDD    Spek   
    • Andreas Scherbaum
    Virtual memory settings in Linux - The Problem with Overcommit How to tune the Memory Overcommit settings in Linux Categories:   Linux    Greenplum Database    Virtual Memory    Overcommit   
    • Lucas Lonergan
    Running GPDB using Docker A look into how the GPDB R&D team uses Docker to increase development consistency. Categories:   GPDB    Docker   
    • Konstantin Semenov
    • Cassio Dias
    SQL Stored Procedure Versioning Strategy A versioning strategy for SQL stored procedures provides flexibility for developers both on the DB and the application side. Categories:   SQL    Version control   
    • Konstantin Semenov
    • Cassio Dias
    SQL Test Driven Development with Oracle RDBMS Test-driving SQL stored procedures using Oracle SQL Developer IDE. Categories:   SQL    TDD    Oracle   
    • Mark Thomas
    Java Deserialization, JMX and CVE-2016-3427 If you use remote JMX, you need to update your JVM to address CVE-2016-3427 Categories:   Security    Java    Apache Tomcat    Pivotal tc Server   
    • Andreas Scherbaum
    ByteA versus TEXT in PostgreSQL (for textual data) One of our customers switched from MongoDB to PostgreSQL, and the migration tool created all data fields as ByteA instead of TEXT. Makes one wonder, if there is a performance difference and if TEXT could be a wiser choice. Categories:   PostgreSQL    Performance   
    • Danny Berger
    • Brian Cunnie
    • Amin Jamali
    • Shatarupa Nandi
    • Saman Alvi
    How to Deploy a Multi-homed BOSH Director to a vSphere Environment We explore deploying a multi-homed BOSH Director to a vSphere environment to segregate networks in a secure manner. Categories:   BOSH   
    • Sai To Yeung
    TDDing React + Redux Helpful patterns for unit testing a React-Redux app Categories:   TDD    React    Redux    Javascript   
    • Joseph Rodriguez
    Using Postgres to analyze ride data Postgres provides some fantastic functionality to help out with basic data analysis. This article will show you how to generate leaderboards and find streaks in raw sql data. Categories:   PostgreSQL    SQL    Databases   
    • Brenda Chan
    Distributed Pair Programming: What Works! Tales of pair programming on a distributed team. Categories:   Agile    Pair Programming   
    • Matthew Horan
    • Charlie Vieth
    Implementing Containers on Windows: A Deep Dive Technical details for how we implemented containers on Windows for Pivotal Cloud Foundry. Categories:   CF Runtime    Diego    Windows    Containers    .NET   
    • Paul Meskers
    Testing JavaScript's native Promises A straightforward look at how to apply test-driven development to native JavaScript promises. Categories:   TDD    JavaScript    Promises    Testing   
    • Brian Watkins
    Faking OAuth2 Single Sign-on in Spring, Two Ways When your Java Spring web application depends on a third-party OAuth2 single sign-on service, tests can be slow, brittle, or difficult to control. I’ll describe two ways to address these issues by faking OAuth2 single sign-on in your tests. Categories:   Spring    Java    Testing   
    • Edwin Xie
    • Corey Innis
    Running Tests in AWS Lambda Quickly and easily run your tests on AWS without the hassle of starting new EC2 instances. Categories:   AWS Lambda    Testing   
    • Damien Le Berrigaud
    • Jack Coy
    ETL Journey from Oracle to Postgres How we transferred a legacy Oracle database to a new Postgres database in a 3 hour window. Categories:   ETL    oracle    postgres    database   
    • Andreas Scherbaum
    SERIAL Datatype Performance in Greenplum Database How to improve the performance of the SERIAL datatype in Greenplum Database Categories:   PostgreSQL    Greenplum Database    Databases    Datatypes    SERIAL   
    • Tira Odhner
    "Some Blog Post" or, How I Learned to Stop Worrying and Like Red Junit Tests Tips and tricks for writing tests that fail well. What to mock, what to name your tests, and how to when. Categories:   Java    Testing   
    • Srivatsan Ramanujam
    Building machine learning models at scale for data parallel problems on Pivotal's MPP databases Building machine learning models (ex: scikit-learn) at scale for data parallel problems on Pivotal’s MPP databases (Greenplum/HAWQ). Categories:   Data Science    Greenplum    Procedural Languages    Python   
    • Mike Dalessio
    Making A Useful C++ Buildpack A useful C++ buildpack needs to consider header files and libraries, not just make. Here’s a story about how I made a useful buildpack for a C++ web framework. Categories:   CF Runtime   
    • Mike Gehard
    Algebraic Data Types In Kotlin Getting feedback quickly about mistakes in your code is a key tenet of agile development. This article will show you how to use algebraic data types and the Kotlin compiler to get fast feedback when you have missed handling an outcome for a business use case. Categories:   Kotlin    functional programming   
    • Mike Gehard
    The Journey of a Spring Boot application from Java 8 to Kotlin, part 3: Data Classes Kotlin data classes reduce a lot of boilerplate code when it comes to writing POJOs that are used for data exchange. Categories:   Spring Boot    Kotlin    Java   
    • Andreas Scherbaum
    Current TransactionID in Greenplum Database How to find out the current TransactionID in Greenplum Database Categories:   PostgreSQL    Greenplum Database    Databases   
    • Mike Gehard
    The Journey of a Spring Boot application from Java 8 to Kotlin, part 2: Configuration Classes What do Spring Boot configuration classes look like in Kotlin? Categories:   Spring Boot    Kotlin   
    • Andreas Scherbaum
    PgConf.Russia 2016 PgConf.Russia 2016 – Talk: How we made Greenplum Open Source Categories:   PostgreSQL    Conference   
    • Anthony Emengo
    Exploring at Pivotal A candid insight into the adoption of the exploratory testing practice at Pivotal Labs. Categories:   Agile    Exploratory Testing    Charter    CF Runtime   
    • Mike Gehard
    The Journey of a Spring Boot application from Java 8 to Kotlin: The Application Class The first steps along the path of converting a fully functional Java 8/Spring Boot/Spring Cloud application to Kotlin. Categories:   Spring Boot    Kotlin   
    • David Jahn
    Capturing Network Traffic With Docker Containers How to capture and log internet traffic from programs using Docker containers. Categories:   Containers    Docker    Network Traffic Monitoring    Logging & Metrics   
    • Andreas Scherbaum
    PostgreSQL Meetup in Berlin, 2016-01-26 Pivotal hosted a PostgreSQL Meetup in Berlin. Speakers: Andres Freund and Oleksandr Shulgin. Categories:   PostgreSQL    Meetup   
    • Roman Shaposhnik
    Pivotal Data Open Source in 2016: community, community, community! When it comes to Open Source, Pivotal had one kick ass of a year in 2015. Here’s a sneak peak for 2016. Categories:   Open Source    Community    Big Data    Databases   
    • Addison Huddy
    GPORCA, A Modular Query Optimizer, Is Now Open-Source GPORCA has achieved an overall 5X performance improvement across all 99 industry standard benchmark queries. Now we call on the community to help take the project to the next level. Categories:   Big Data    Databases    Query Optimization    SQL   
    • Megha Agarwal
    Pairing for Data Scientists Lets see how pair programming fits in the data science world. Categories:   Data Science    Pair Programming    Agile   
    • Mike Gehard
    Concourse Web Logging You need to debug your Concourse ATC server. How do you turn up the logging level to allow that? Categories:   Concourse   
    • Micah Young
    • John Shahid
    Deploying your first .NET app on Cloud Foundry PCF 1.6 brings with it support for .NET. Here’s how to get started. Categories:   .NET    CF Runtime   
    • Dwayne Forde
    Intro to the Patch Command Quick intro on how to use the patch command to edit, and revert, the text of multiple files. Categories:   Patch    Golang   
    • Mike Dalessio
    Abstraction, or, The Gift of Our Weak Brains Our brains are naturally limited. This can be a curse, or it can be a gift, depending on how you look at it. Categories:   Agile    Humans   
    • Laura Kogler
    Setting up Kotlin with Android and tests First impressions of Kotlin Categories:   Kotlin    Android   
    • Chris Hendrix
    HTTP Trailers Signaling failure during an HTTP stream Categories:   Golang    HTTP    MySQL   
    • Brian Cunnie
    • Rob Dimsdale
    The World's Smallest Concourse CI Server How to deploy a publicly-accessible, extremely lean Concourse CI server. Categories:   BOSH    Concourse   
    • Warren Fernandes
    Using the Cloud Foundry Firehose Plugin Get your Cloud Foundry Firehose logs and metrics straight to your fingertips. Categories:   Firehose    Loggregator    CF CLI    Logging & Metrics   
    • Gareth Smith
    Agile and Program Logic On some of the differences and similarities in perspective between Agile/TDD programmers and developers of program-logic tools. Categories:   Agile   
    • Kai Xiang
    Scaling up to 2000 vms with BOSH In order to know if we can deploy 2000 vms with BOSH, we did a scaling test and this blog post list how we did it and the caveats we found. Categories:   BOSH   
    • Tammer Saleh
    A Team Sport Welcome to our new Engineering Journal!
    • Dan Mikusa
    JVM Tuning for Apache Tomcat Performance Tuning the JVM for Running Apache Tomcat Categories:   Tomcat    Performance    Migrated Content   
    • Dan Mikusa
    Apache Tomcat GC Measurement Setting Up Measurement of Garbage Collection in Apache Tomcat Categories:   Tomcat    Performance    Migrated Content   
    • Filip Hanik
    Apache Tomcat jdbc-pool Configuration and use of Apache Tomcat’s high concurrency database connection pool Categories:   Tomcat    Database    JDBC Pool    Migrated Content